U e{j|@sdZddlZddlZddlZddlmZddlmZddlZ ddl Z ddl Z ddl m ZddlmZddlmZmZddlmZmZmZdd lmZmZmZdd lmZdd lmZdd l m!Z!dd l m"Z#ddl$m%Z%ddddddddddddddgZ&e'e drej(dksej)dkr&e&*deZ+e&D]pZ,e,dkrPe -d d!d gZ.n*e,dkrle -d"d#d$gZ.ne j/d%e,d&Z.ee.d'd(e,d)e,did*Z0e0e+e,<q0d+d,ge+j1d-<d.d/d0d1d2d3d4d5d6d7d8g Z2d9d:Z3d;d<Z4d=d>Z5d?d@Z6dAdBZ7dCdDZ8dEdFZ9dGdHZ:dIdJZ;dKdLZdQdRZ?dSdTZ@e jABdUeefdVdWZCdXdYZDe jABdZeEeFe jABdUeefe jABd[d\d]d^ZGd_d`ZHdadbZIe jABdUeefdcddZJdedfZKe jABdgdhdige jABdje jLe jMeNeOge jABdkdldmgdndoZPe jABdpdqdrgdsdtZQdudvZRdwdxZSdydzZTd{d|ZUd}d~ZVddZWddZXddZYddZZddZ[iZ\iZ]e j-ddge j^d&e]d<dddge\d<e j_de`de]d<Z0ddgddgge0d<ddd%gdddgge0d<ddddge\d<e j_de`de]d<Z0e j-ddgd%dgge jad&e0d<e j-ddd%gdddgge jad&e0d<ddddge\d<e j-ddgd%ddgge`d&e]d<ddddge\d<e j-e j-ddge jbd&e j-d%ddge jbd&ge`d&e]d<ddddge\d<e j-e -ddge -dddgge`d&e]d<ddddge\d<e j-e -d!d ge -d!d d!gge`d&e]d<ddddge\d<e j-ddgd%dgge jcd&e]d<ddddge\d<e j/de jdd&eddde]d<ddddge\d<e j-ddiddige`d&e]d<ddddge\d<e j-ddiddigddiddigge`d&e]d<ddddge\d<e jABdeEefe]e]ge\gddZhddZiddZjddZkddZldS)zN This module tests some of the methods related to the ``ECSV`` reader/writer. N) nullcontext)StringIO)units)ascii) DELIMITERSInvalidEcsvDatatypeWarning) compare_attrs mixin_colsserialized_names)ColumnQTableTable) MaskedColumn) simple_table) QuantityInfo)allclose)TEST_DIRboolint8int16int32int64uint8Zuint16Zuint32Zuint64float16float32float64Zfloat128strntlFTzab 0zab, 1Zab2dtypezm / sZdescr_zmeta )unit descriptionmetaZcomment1Zcomment2comments # %ECSV 1.0# --- # datatype:z# - {name: a, datatype: int64}z # - {name: b, datatype: float64}z# - {name: c, datatype: string}# schema: astropy-2.0a b cz1 1.0 cz2 2.0 dz3 3.0 ecCs2t}t}|j|dd|tks.tdS)z Write a simple table with common types. This shows the compact version of serialization with one line per column. ascii.ecsvformatN)rrwritegetvalue splitlines SIMPLE_LINESAssertionErrortoutr6_/var/www/html/bwave/python/venv/lib/python3.8/site-packages/astropy/io/ascii/tests/test_ecsv.pytest_write_simplePsr8cCsttd}dddddddd d dd d d dddddddddddddddddg}t}|j|dd||ksptdS) zV Write a full-featured table with common types and explicitly checkout output )rrrrr&r'r(z# - name: boolz# unit: m / sz# datatype: boolz# description: descr_boolz# meta: {meta bool: 1}z# - name: int64z# datatype: int64z# description: descr_int64z# meta: {meta int64: 1}z# - name: float64z# datatype: float64z# description: descr_float64z# meta: {meta float64: 1}z # - name: strz# datatype: stringz# description: descr_strz# meta: {meta str: 1}z# meta: !!omapz"# - comments: [comment1, comment2]r)zbool int64 float64 strzFalse 0 0.0 "ab 0"zTrue 1 1.0 "ab, 1"zFalse 2 2.0 ab2r+r,N)T_DTYPESrr.r/r0r2)r4linesr5r6r6r7test_write_full\sF!r;c Cst}tD]}t}|j|d|dtj|ddtj|ddt|tj|dddtj|ddg}|D]P}|j|jkst |j D]4}|| ||st t ||||kst qqxqdS) z Write a full-featured table with all types and see that it round-trips on readback. Use both space and comma delimiters. r+r- delimiterr,recsvFr-guessN)r9rrr.r readr/rr$r2colnamesZ attrs_equalnpall)r4r=r5Zt2st2namer6r6r7test_write_read_roundtrips  rGc CsBt}tt}tj|dddW5QRXdt|jks>tdS)zJ Passing a delimiter other than space or comma gives an exception r+|r< only space and comma are allowedN) rpytestraises ValueErrorr9r.rvaluer2)r5errr6r6r7test_bad_delimiters rOc CsDtt}d|d<ttjtjd|dddW5QRXdS)z0 Bad header without initial # %ECSV x.x z # %ECV 0.9r r+Fr?N) copyr1rJrKrZInconsistentTableErrorr rAjoin)r:r6r6r7test_bad_header_starts rSc CsXtt}|ddtt}tjd|dddW5QRXdt |j ksTt dS) z$ Illegal delimiter in input z# delimiter: |rPr+Fr?rIN) rQr1insertrJrKrLr rArRrrMr2)r:rNr6r6r7test_bad_delimiter_inputs     rVcCst}tdddd|d<d|dj_ddi|dj_ddg|d<t}|j|d d tj | d d }t |d|dkst |dj |dj kst |dj|djkst |djj|djjkst |djj|djjkst t |d|dks t d S) z+ Multi-dimensional column in input rTrar#rbr+r,N)r rCarangereshapeinfor#r$rr.rAr/rDr2shaper!r4r5rEr6r6r7test_multidim_inputs  r`cCs\t}tddddgffddddgffgd d d d gfgd ddddid|d<tddgddggdtdd|d<t}|j|ddtj|dd}|jD]}t ||||kst ||j ||j kst ||j ||j kst ||j||jkst ||j||jkst ||jj||jjks:t ||jj||jjkst qdS)z% Structured column in input. B?@@A"@ @@)sZU1v)p0Zf8)p1Z2f8r#>mrrT)r!r#r-r"r$rY)rbrc)rfrg)rdg@)rhg@zf8,f8zm,s)r!r"rZr+r,N)r r uZUnitrr.rAr/rBrCrDr2r^r!r"r-r]r#r$)r4r5rEcolr6r6r7test_structured_inputs2   rqcCsbttddgdddgd}t}|j|ddtj|dd}|j|jksNtt|d ks^td S) zLTest fix in #5010 for issue #5009 (ECSV fails for empty type with bool type)ifrYrZc)r!namesr+r,rN) r rrr.rAr/r!r2lenr_r6r6r7test_round_trip_empty_tables rwc CsVtt}|d}d||<tt}tj|ddW5QRXdt|j ksRt dS)z{ Test that mismatch in column names from normal CSV header vs. ECSV YAML header raises the expected exception. r*za b dr>r,z-column names from ECSV header ['a', 'b', 'c']N) rQr1indexrJrKrLrrArrMr2)r:Z header_indexrNr6r6r7test_csv_ecsv_colnames_mismatchs    rycCsbt}dtjtjd|_dgtj|d<t}|j|ddd|ksNtd|ks^td S) zF See https://github.com/astropy/astropy/issues/5604 for more. )ZfooZfoo2barr+r,z!astropy.units.Unitz!astropy.units.QuantityN) r rokmrir$rr.r/r2r3r6r6r7test_regression_5604sr~c Cs$|r|j|jkst|j|jks$tdddddg}||D]}|}|}|dD]B}zt||}t||}WqPtk r||}||}YqPXqPt|tjr|j j dkrt ||dd stq:t ||ks:tq:|s t|tjr |j j dkr t ||d d s tnt ||ks tdS) Nz info.namez info.formatz info.unitzinfo.descriptionz info.dtype.rs绽|=ZrtolgV瞯<) __class__r2r^splitgetattrAttributeError isinstancerCndarrayr!kindquantity_allcloserD) Zobj1obj2attrs compare_classZ info_attrsattrZa1Za2Zsubattrr6r6r7assert_objects_equals4  rcCstddtD}t}|j|ddtj|dd}t|t ksLt td|d<t}|j|ddtj|dd}t|tkst dS)zzEnsure that ascii.read(ecsv_file) returns the correct class (QTable if any Quantity subclasses, Table otherwise). cSs"i|]\}}t|jts||qSr6)rr]r).0rFrpr6r6r7 Hs z5test_ecsv_mixins_ascii_read_class..r+r,r>ZlonN) r r itemsrr.rrAr/typer r2r_r6r6r7!test_ecsv_mixins_ascii_read_classBs rc Cstt}tdd|D|d}t}|j|ddtj|dd}|j|jksTt |j D]f\}}||}t |}d}t |jtrt|tkst dg}d}tj|j|d d st t||||q^d S) zWTest writing as QTable and reading as Table. Ensure correct classes come out. cSsg|] }t|qSr6r rrFr6r6r7 bsz4test_ecsv_mixins_qtable_to_table..rur+r,Tr"FrrN)sortedr r rr.r rAr/rBr2columnsrrrr]rrr rCrrMr) rur4r5rErFrpZcol2rrr6r6r7 test_ecsv_mixins_qtable_to_table\s  r table_clsc Cstt}g}|D].}t|}|ds4dd|D}||q|dd|D|d}t}|j|dd|j|dd}|j |j kst |j|dd}|j |kst d S) zGTest write/read all cols at once and validate intermediate column namesZtm3cSs"g|]}|ds|ddqS)jd2z.jd1)endswithreplace)rZs_namer6r6r7rs z+test_ecsv_mixins_as_one..cSsg|] }t|qSr6rrr6r6r7rsrr+r,z ascii.basicN) rr r startswithextendrr.rAr/rBr2) rruZall_serialized_namesrFZs_namesr4r5rEt3r6r6r7test_ecsv_mixins_as_onezs   rcCsN|dkrJddl}ddt|ddgtd|D}||dg|}|S)a5Take a col with length=2 and make it N-d by repeating elements. For the special case of ndim==1 just return the original. The output has shape [3] * ndim. By using 3 we can be sure that repeating the two input elements gives an output that is sufficiently unique for the multidim tests. rrNcSsg|] \}}|qSr6r6)ridx_r6r6r7rsz!make_multidim..r) itertoolszipcycleranger\)rpndimrZidxsr6r6r7 make_multidims &rname_colr)rrTrc s |\}}ttddg|}t||}|||gd|dgd}d||j_t}|j|ddj|dd}|j |j kst |j D]T} t || j |kst | d krd g} nfd d t | D} t|| || | q|d r||jjjtjkst ||jjjtjkst dS)z~Test write/read one col at a time and do detailed validation. This tests every input column type as 1-d, 2-d and 3-d. rbrcc1c2rr#r+r,)rrdatacs g|]}|dkrtks|qS)Z wrap_angle)r )rrrr6r7rsz/test_ecsv_mixins_per_column..tmN)rrCarrayr]r#rr.rAr/rBr2rvr^rrr_timeZjd1rrr) rrrrFrprtr4r5rEZcolnamecomparer6rr7test_ecsv_mixins_per_columns(     rcCs|d}tdd}||t|}|jdks4t|j|jksDt|jD]j}t||j ||j ksltt||||kstd||_ d||_ t||||krJtqJdS)a_Test (mostly) round-trip of MaskedColumn through ECSV using default serialization that uses an empty string "" to mark NULL values. Note: >>> simple_table(masked=True) a b c int64 float64 str1 ----- ------- ---- -- 1.0 c 2 2.0 -- 3 -- e test.ecsvTmaskedFN rr.r rArr2rBrCrDmasktmp_pathfilenamer4rErFr6r6r7$test_round_trip_masked_table_defaults       rcCs|d}tdd}d|dd<ddd g|d <|j|d d t|}|jd ksRt|j|jksbt|jD]j}t||j ||j kstt||||kstd ||_ d ||_ t||||kshtqhdS)zY Same as prev but set the serialize_method to 'data_mask' so mask is written out rTrrrtrrrTrd data_mask)serialize_methodFNrrr6r6r7+test_round_trip_masked_table_serialize_masks      rc Cs||d}td}|}td||d<|||tkrNtjtjdd}nt }|| |}W5QRXt |dj tj stt|dj dkst|tkrt|dj|djkstnt|d|dkstt|| |}|dj |kstt|d|dks"t|d}||| |} | dj |ksRtt| d|dksntW5QRXdS) z8Ensure that we can read-back enabled user-defined units.rZbandpass_sol_lumrzlz#'bandpass_sol_lum' did not parse .*matchz test2.ecsvN)roZdef_unitrCr[r.r rJwarnsZ UnitsWarningrrArr"ZUnrecognizedUnitr2rrDrMZadd_enabled_units) rrrr"r4ctxrEr filename2Zt4r6r6r7&test_ecsv_round_trip_user_defined_unit s0       rcCsfd}tj|dd}|d}t|ts(tt|jdddddgksFtt|dddddgksbtdS)Nzl# %ECSV 1.0 # --- # datatype: # - {name: col0, datatype: bool} # schema: astropy-2.0 col0 1 0 True "" False r>r,Zcol0FT)rrArrr2rCrDr)txtZdatrpr6r6r7test_read_masked_bool5s  rrZ null_valuerr!r=, cCst}ttdddd|d}|tkr4|d}d|jd<d|jd<||d <d d g|d <t}|j |d |dtj | d d}|j dkst |j|jkst |jD]b}||j||jkst t||drt||j||jkst t||||kst qdS)N rTrr ZU2T)rrr)rrrrYxyrZr+)r-rr,Fr)r rrCr[r\rZastyperrr.rAr/rr2rBr!hasattrrD)rr!r=r4rpr5rErFr6r6r7$test_roundtrip_multidim_masked_arrayJs$     rsubtypezsome-user-typecomplexc Cshd|d}tjtd|ddtj|dd}W5QRX|djjd ksPt|dd d ksdtd S) z8Test an ECSV file with a string type but unknown subtypezL# %ECSV 1.0 # --- # datatype: # - name: a # datatype: string # subtype: z$ # schema: astropy-2.0 a [1,2] [3,4]zunexpected subtype 'z' set for column 'a'rr>r,rYUrz[1,2]N)rJrrrrAr!rr2)rrr4r6r6r7test_multidim_unknown_subtypees  rc Cs0d}tjtddtj|ddW5QRXdS)Test a malformed ECSV filezx# %ECSV 1.0 # --- # datatype: # - name: a # datatype: string # subtype: int64[3] # schema: astropy-2.0 a [1,2] [3,4]z,column 'a' failed to convert: shape mismatchrr+r,NrJrKrLr rA)rr6r6r7test_multidim_bad_shape}s  rc CsVt}tjddhdgtd|d<d}t}tjt|d|j|ddW5QRXdS) NrrTr rYzScould not convert column 'a' to string: Object of type set is not JSON serializablerr+r,) r rCrobjectrrJrK TypeErrorr.)r4rr5r6r6r7 test_write_not_json_serializablesrc Cs4d}d}tjt|dtj|ddW5QRXdS)rzo# %ECSV 1.0 # --- # datatype: # - {name: a, datatype: string, subtype: json} # schema: astropy-2.0 a fail [3,4]z.cSs"g|]}|dr|ddqS)#rTN)rrr6r6r7rs rTrP)r0yamlZ safe_loadrR)textr:r5r6r6r7_get_ecsv_header_dict s  rcCsddlm}|D]r\}}t}|||<t}|j|ddt|}||d}td|d|ddt|ddtqdS) Nr)pformatr+r,datatypezexps[z] =r) pprintrrr rr.rr/print)colsrrFrpr4r5hdrZfmt_hdrr6r6r7_make_expected_valuess   rrTZscalar)rrF)rT)r^r!z2-d variable array listsrYrXg@stringjson)rrFrz2-d variable array numpyg@zfloat32[2,null]rzz1-d variable array listsz1-d variable array numpyz uint8[null]rZrtrez1-d variable array numpy strz string[null]z1-d variable array numpy boolz bool[null]z1-d regular arrayzint8[2]z2-d regular arrayz float16[2,2]z scalar objectz 1-d objectzjson[2]z name,col,expc Cst}|||<t}|j|ddt|}|d|ks>ttj|dd}|j|jks`t|jD]f}||j||jkstt ||||D]6\}}t |t j r|j|jkstt ||kstqqfdS)z=Test variable length lists, multidim columns, object columns.r+r,rN)r rr.rr/r2rArBr!rrrCrrD) rFrpexpr4r5rrEval1val2r6r6r7test_specialized_columns|s   rc Cs@ttjtdd}dd}|j|ks.tddddd d d d d }| D]}|j }|j dkrbqLt |t spt|j dd}|j dd}|dkr|jdkst|dkr|jdkst|dkr$|jdkst|jj dkst|D]<}t |tjst|jj ||stt|dkstqqL|jj ||sLtqLdS)aORead ECSV file created by M. Taylor that includes scalar, fixed array, variable array for all datatypes. This file has missing values for all columns as both per-value null and blank entries for the entire column value. Note: original file was modified to include blank values in f_float and f_double columns. rz subtypes.ecsvzi_index,s_byte,s_short,s_int,s_long,s_float,s_double,s_string,s_boolean,f_byte,f_short,f_int,f_long,f_float,f_double,f_string,f_boolean,v_byte,v_short,v_int,v_long,v_float,v_double,v_string,v_boolean,m_int,m_doublerrrrrrrrr)byteshortintlongfloatdoublerbooleanZi_indexrTNrri)rs)rrrjr)rrrTr)r rAospathrRrrrBr2Zitercolsr]rFrrr^r!rCrrrv)r4rBZtype_maprpr] type_namervalr6r6r7test_full_subtypessD    r cCsd}tj|dd}t|dtjddddggtd kssR         $      +   "   %  $! '    $  (            9%