CoderFunda
  • Home
  • About us
    • Contact Us
    • Disclaimer
    • Privacy Policy
    • About us
  • Home
  • Php
  • HTML
  • CSS
  • JavaScript
    • JavaScript
    • Jquery
    • JqueryUI
    • Stock
  • SQL
  • Vue.Js
  • Python
  • Wordpress
  • C++
    • C++
    • C
  • Laravel
    • Laravel
      • Overview
      • Namespaces
      • Middleware
      • Routing
      • Configuration
      • Application Structure
      • Installation
    • Overview
  • DBMS
    • DBMS
      • PL/SQL
      • SQLite
      • MongoDB
      • Cassandra
      • MySQL
      • Oracle
      • CouchDB
      • Neo4j
      • DB2
      • Quiz
    • Overview
  • Entertainment
    • TV Series Update
    • Movie Review
    • Movie Review
  • More
    • Vue. Js
    • Php Question
    • Php Interview Question
    • Laravel Interview Question
    • SQL Interview Question
    • IAS Interview Question
    • PCS Interview Question
    • Technology
    • Other

08 April, 2024

pandas/pyarrow ArrowTypeError: Unable to merge: Field on handcrafted partitioned parquet

 Programing Coderfunda     April 08, 2024     No comments   

I read the parquet file directly
pd.read_parquet(r'C:\Datasets\cn_data\dm\qmt\wqa_mfeatures\30m\year=2020\month=11\data.parquet')



No error will be reported.


But when I read the directory:
pd.read_parquet(r'C:\Datasets\cn_data\dm\qmt\wqa_mfeatures\30m\year=2020')



An error will be reported
ArrowTypeError: Unable to merge: Field month has incompatible types: int32 vs dictionary



This is because I handcrafted this partitioned path.
It is important that I have to hand craft the partitioned path.
I have 5000 item need transform and write to df.to_parquet(path, partition_cols=['year', 'month']) , and yes it would not overwrite existing files.



* But if I only need rerun 300 item, it would preduce new files, I can't delete the data produce by last run.

* With time goes, I need rerun transform function on new dates(year,month), I need new data overwrite old data.






I just want to keep pd.read_parquet function working with handcraft paths, this can reduce many works on reimplement a similar stuff and refactor pd.read_parquet in many projects.
  • Share This:  
  •  Facebook
  •  Twitter
  •  Google+
  •  Stumble
  •  Digg
Email ThisBlogThis!Share to XShare to Facebook

Related Posts:

  • Laravel + join 2 eloquent queries “laravel + join 2 eloquent queries” Coderfunda Code Answerlaravel + join 2 eloquent queries11 $select->joinSub(2 $sel… Read More
  • Laravel join “laravel join” Coderfunda Code Answer’slaravel join231Inner Join : ->join('contacts', 'users.id', '=', 'contacts.user_id')2Left Join … Read More
  • Join in laravel eloquent “join in laravel eloquent” Coderfunda Code Answer’slaravel join231Inner Join : ->join('contacts', 'users.id', '=', 'contacts.user_id')2… Read More
  • Laravel join query sum example“laravel join query sum example” Coderfunda Code Answer’ssql select sum group by id laravel join01$users = User::select('users*', 'analytics.*', … Read More
  • Laravel raw query join many to many “laravel raw query join many to many” Coderfunda Code Answerlaravel raw query join many to many01$query = DB::table('blog_posts')2->join… Read More
Newer Post Older Post Home

0 comments:

Post a Comment

Thanks

Meta

Popular Posts

  • Vue3 :style backgroundImage not working with require
    I'm trying to migrate a Vue 2 project to Vue 3. In Vue 2 I used v-bind style as follow: In Vue 3 this doesn't work... I tried a...
  • SQL ORDER BY Keyword
      The SQL ORDER BY Keyword The ORDER BY keyword is used to sort the result-set in ascending or descending order. The ORDER BY keyword sorts ...
  • Enabling authentication in swagger
    I created a asp.net core empty project running on .net6. I am coming across an issue when I am trying to enable authentication in swagger. S...
  • failed to load storage framework cache laravel excel
       User the export file and controller function  ..         libxml_use_internal_errors ( true ); ..Good To Go   public function view () : ...
  • Features CodeIgniter
    Features CodeIgniter There is a great demand for the CodeIgniter framework in PHP developers because of its features and multiple advan...

Categories

  • Ajax (26)
  • Bootstrap (30)
  • DBMS (42)
  • HTML (12)
  • HTML5 (45)
  • JavaScript (10)
  • Jquery (34)
  • Jquery UI (2)
  • JqueryUI (32)
  • Laravel (1017)
  • Laravel Tutorials (23)
  • Laravel-Question (6)
  • Magento (9)
  • Magento 2 (95)
  • MariaDB (1)
  • MySql Tutorial (2)
  • PHP-Interview-Questions (3)
  • Php Question (13)
  • Python (36)
  • RDBMS (13)
  • SQL Tutorial (79)
  • Vue.js Tutorial (68)
  • Wordpress (150)
  • Wordpress Theme (3)
  • codeigniter (108)
  • oops (4)
  • php (853)

Social Media Links

  • Follow on Twitter
  • Like on Facebook
  • Subscribe on Youtube
  • Follow on Instagram

Pages

  • Home
  • Contact Us
  • Privacy Policy
  • About us

Blog Archive

  • September (100)
  • August (50)
  • July (56)
  • June (46)
  • May (59)
  • April (50)
  • March (60)
  • February (42)
  • January (53)
  • December (58)
  • November (61)
  • October (39)
  • September (36)
  • August (36)
  • July (34)
  • June (34)
  • May (36)
  • April (29)
  • March (82)
  • February (1)
  • January (8)
  • December (14)
  • November (41)
  • October (13)
  • September (5)
  • August (48)
  • July (9)
  • June (6)
  • May (119)
  • April (259)
  • March (122)
  • February (368)
  • January (33)
  • October (2)
  • July (11)
  • June (29)
  • May (25)
  • April (168)
  • March (93)
  • February (60)
  • January (28)
  • December (195)
  • November (24)
  • October (40)
  • September (55)
  • August (6)
  • July (48)
  • May (2)
  • January (2)
  • July (6)
  • June (6)
  • February (17)
  • January (69)
  • December (122)
  • November (56)
  • October (92)
  • September (76)
  • August (6)

  • Failed to install 'cordova-plugin-firebase': CordovaError: Uh oh - 9/21/2024
  • pyspark XPath Query Returns Lists Omitting Missing Values Instead of Including None - 9/20/2024
  • SQL REPL from within Python/Sqlalchemy/Psychopg2 - 9/20/2024
  • MySql Explain with Tobias Petry - 9/20/2024
  • How to combine information from different devices into one common abstract virtual disk? [closed] - 9/20/2024

Laravel News

  • Clean Up Your Code with the whenHas Method - 6/5/2025
  • Laravel OpenRouter - 6/4/2025
  • Enable Flexible Pattern Matching with Laravel's Case-Insensitive Str::is Method - 5/31/2025
  • Cast Model Properties to a Uri Instance in 12.17 - 6/4/2025
  • Simplify Negative Relation Queries with Laravel's whereDoesntHaveRelation Methods - 5/31/2025

Copyright © 2025 CoderFunda | Powered by Blogger
Design by Coderfunda | Blogger Theme by Coderfunda | Distributed By Coderfunda