Feature/auto [infer-schema & auto-directory ingestion]

[mccrearyd]

made it very easy to write dataset auto ingestion code. this branch supports image_classification directories that are structured as class_folder -> images.

it also removed the from_directory method on hub.Dataset & replaced it with from_path.

[github-actions]

Locust summary

Git references

Initial: 1a62a6c
Terminal: 9cef57c

---

hub/auto/util.py

Changes:

• Name: DirectoryParserState
  Type: class
  Changed lines: 32
  Total lines: 32
  Changes:
  • Name: DirectoryParserState.__init__
    Type: function
    Changed lines: 3
    Total lines: 3
  • Name: DirectoryParserState.get_parsers
    Type: function
    Changed lines: 7
    Total lines: 7
  • Name: DirectoryParserState.directory_parser
    Type: function
    Changed lines: 19
    Total lines: 19
    Changes:
• Name: get_children
  Type: function
  Changed lines: 4
  Total lines: 4
• Name: get_image_shape
  Type: function
  Changed lines: 5
  Total lines: 5
• Name: get_ext
  Type: function
  Changed lines: 3
  Total lines: 3
• Name: files_are_of_extension
  Type: function
  Changed lines: 11
  Total lines: 11

hub/api/dataset.py

Changes:

• Name: Dataset
  Type: class
  Changed lines: 4
  Total lines: 943
  Changes:
  • Name: Dataset.__init__
    Type: function
    Changed lines: 1
    Total lines: 183
  • Name: Dataset.from_path
    Type: function
    Changed lines: 3
    Total lines: 4

hub/auto/computer_vision/classification.py

Changes:

• Name: image_classification
  Type: function
  Changed lines: 67
  Total lines: 67
  Changes:
  • Name: image_classification.upload_data
    Type: function
    Changed lines: 6
    Total lines: 6

hub/auto/infer.py

Changes:

• Name: _find_root
  Type: function
  Changed lines: 29
  Total lines: 29
• Name: infer_dataset
  Type: function
  Changed lines: 35
  Total lines: 35

hub/auto/tests/test_image_classification.py

Changes:

• Name: assert_conversion
  Type: function
  Changed lines: 26
  Total lines: 26
• Name: get_dataset_store
  Type: function
  Changed lines: 3
  Total lines: 3
• Name: _exec_command
  Type: function
  Changed lines: 7
  Total lines: 7
• Name: _download_kaggle
  Type: function
  Changed lines: 14
  Total lines: 14
• Name: test_dandelionimages
  Type: function
  Changed lines: 3
  Total lines: 3
• Name: test_flowers_recognition
  Type: function
  Changed lines: 7
  Total lines: 7

[mccrearyd]

the tests i wrote may fail for windows... hopefully not though

EDIT: aha! it didn't >:)

[mccrearyd]

don't merge yet, i added some asserts that should fail, if they don't then there might be a problem. basically the tests before should not have passed because kaggle credentials are required to run.

[mynameisvinn]

I think many image datasets follow coco or voc style annotations, should we take that into account?

[mccrearyd]

I think many image datasets follow coco or voc style annotations, should we take that into account?

yes, but this is version 1. i don't care to support all datasets right off the bat. this is to get the feature into the hands of testers immediately so we can iterate. there will be many followup PRs.

[mccrearyd]

this is basically ready to merge, i rewrote the tests to not use kaggle datasets. but only 1 test failed and its outside my context (the master branch clci is failing). should we merge it anyways? @mynameisvinn @AbhinavTuli @imshashank

source: https://app.circleci.com/pipelines/github/activeloopai/Hub/2089/workflows/6f462b61-3e37-4529-a88a-73c55fe6d94a/jobs/3082

[codecov]

Codecov Report

Merging #696 (fb59b54) into master (c08b6af) will increase coverage by 0.14%.
The diff coverage is 91.11%.

@@            Coverage Diff             @@
##           master     #696      +/-   ##
==========================================
+ Coverage   89.09%   89.23%   +0.14%     
==========================================
  Files          58       63       +5     
  Lines        4299     4366      +67     
==========================================
+ Hits         3830     3896      +66     
- Misses        469      470       +1     

Impacted Files	Coverage Δ
hub/auto/infer.py	82.85% <82.85%> (ø)
hub/auto/computer_vision/classification.py	92.15% <92.15%> (ø)
hub/auto/util.py	95.34% <95.34%> (ø)
hub/api/dataset.py	91.70% <100.00%> (+2.12%)	⬆️
hub/auto/__init__.py	100.00% <100.00%> (ø)
hub/auto/computer_vision/__init__.py	100.00% <100.00%> (ø)
hub/store/metastore.py	85.04% <0.00%> (-5.61%)	⬇️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c08b6af...fb59b54. Read the comment docs.

[mccrearyd]

nice i see you fixed the tests! @AbhinavTuli good stuff