Start duckdb implementation of dremio

[ahuang11]

Started migrating stuff over to DuckDB source and will start testing.

Based off https://www.dremio.com/blog/using-duckdb-with-your-dremio-data-lakehouse/

With a well-curated semantic layer, Dremio makes access and discovery of data much easier for analysts to pull the data they need and have permission to access using Apache Arrow Flight. They can then proceed locally for ad hoc queries on the subset of the data they pulled from Dremio

So essentially, we still have to read the entire dataset and migrate it into a duckdb source. Previously with intake-dremio, I think we were doing dremio -> intake -> duckdb. now it's dremio -> duckdb

While doing that, would like your feedback on implementation or anything.

I think there are things we can tinker with for optimizing the reads like read_chunk and partitioning, etc

def dataset(source, schema=None, format=None, filesystem=None,
            partitioning=None, partition_base_dir=None,
            exclude_invalid_files=None, ignore_prefixes=None):

[ahuang11]

Unsure where I should put this; I pulled it from intake-dremio.

[codecov]

Codecov Report

Attention: Patch coverage is 16.05839% with 115 lines in your changes missing coverage. Please review.

Project coverage is 56.22%. Comparing base (2622c0b) to head (7d93365).
Report is 8 commits behind head on main.

Files	Patch %	Lines
lumen/sources/duckdb.py	20.95%	83 Missing ⚠️
lumen/sources/dremio_utils.py	0.00%	32 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #683      +/-   ##
==========================================
- Coverage   57.03%   56.22%   -0.81%     
==========================================
  Files          98       99       +1     
  Lines       11598    11804     +206     
==========================================
+ Hits         6615     6637      +22     
- Misses       4983     5167     +184     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.