{"id":2224,"date":"2017-12-27T11:44:48","date_gmt":"2017-12-27T09:44:48","guid":{"rendered":"http:\/\/soci.hu\/blog\/?p=2224"},"modified":"2017-12-27T11:44:48","modified_gmt":"2017-12-27T09:44:48","slug":"python-pandas-lassu-io-parsers-read_csv-metodus","status":"publish","type":"post","link":"https:\/\/soci.hu\/blog\/index.php\/2017\/12\/27\/python-pandas-lassu-io-parsers-read_csv-metodus\/","title":{"rendered":"Python pandas lass\u00fa io.parsers.read_csv met\u00f3dus"},"content":{"rendered":"

Elkezdtem pythonozni, mivel machine learninget tanulok, \u00e9s ahhoz vagy python vagy R javasolt. Legjobb mindkett\u0151h\u00f6z \u00e9rteni, most a pyhon van soron.<\/p>\n

Akit \u00e9rdekel b\u00e1tran v\u00e1gjon bele, a nyelv nem nagy sz\u00e1m (nekem ronda ez a kett\u0151spontos mindens\u00e9g, de majd megszokom), a tanul\u00e1st a libek megismer\u00e9se viszi el (kb. mint a legt\u00f6bb nyelvn\u00e9l).<\/p>\n

Mivel van sok t\u0151zsdei adatom ezeken futtatom az ML libeket. A legt\u00f6bb p\u00e9ld\u00e1ban napos adatokat haszn\u00e1lnak, de \u00e9n intraday akarok keresg\u00e9lni, ami sokkal t\u00f6bb adatot jelent. Val\u00f3sz\u00edn\u0171leg bajban leszek, pl. a Support Vector Machine o(n^4)-es alg, quadratikus, \u00edgy nem tolhatok r\u00e1 t\u00fal sok adatot.
\nDe m\u00e1r az elej\u00e9n elakadtam, mert 1-2M CSV bet\u00f6lt\u00e9se is 10mp volt.<\/p>\n

Mindenf\u00e9l\u00e9ket \u00edrtak a stackoverflown, csak a megold\u00e1st nem.<\/p>\n

\r\nsym1 = pd.io.parsers.read_csv(os.path.join(datadir, '%s.txt' % symbol),\r\nheader=None, index_col=0, nrows = rows, parse_dates=[['Date', 'Time']],\r\ninfer_datetime_format=True,\r\nnames=['Date', 'Time','Open','High','Low','Close', 'Up', 'Down'], usecols=['Date', 'Time','Close'])\r\n<\/pre>\n
Az infer_datetime_format=True hozott megold\u00e1st, valami\u00e9rt a d\u00e1tum parsol\u00e1sa csapnival\u00f3, ezen flag n\u00e9lk\u00fcl. A read_csv doksija \u00edrta, onnan j\u00f6tt az \u00f6tlet:
\n“infer_datetime_format : boolean, default False<\/p>\n
If True and parse_dates is enabled, pandas will attempt to infer the format of the datetime strings in the columns, and if it can be inferred, switch to a faster method of parsing them. In some cases this can increase the parsing speed by 5-10x.”<\/p>\n","protected":false},"excerpt":{"rendered":"
Elkezdtem pythonozni, mivel machine learninget tanulok, \u00e9s ahhoz vagy python vagy R javasolt. Legjobb mindkett\u0151h\u00f6z \u00e9rteni, most a pyhon van soron. Akit…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[97,4,96],"tags":[],"class_list":["post-2224","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-szakmai-elet","category-python"],"_links":{"self":[{"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/2224","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/comments?post=2224"}],"version-history":[{"count":1,"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/2224\/revisions"}],"predecessor-version":[{"id":2225,"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/posts\/2224\/revisions\/2225"}],"wp:attachment":[{"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/media?parent=2224"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/categories?post=2224"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/soci.hu\/blog\/index.php\/wp-json\/wp\/v2\/tags?post=2224"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}