diff --git a/docs/courses/data-science/2.1-single-objective.md b/docs/courses/data-science/2.1-single-objective.md index 9010958..8534595 100644 --- a/docs/courses/data-science/2.1-single-objective.md +++ b/docs/courses/data-science/2.1-single-objective.md @@ -1,21 +1,42 @@ # 🧩 2.1 Single-objective Optimization -% ```{contents} -% :depth: 2 -% ``` +##�� Learning Objectives +- Understand the fundamentals of Bayesian Optimization (BO) +- Learn to implement single-objective optimization using Honegumi +- Master practical aspects of experimental optimization +- Develop skills in model validation and analysis -## 🔰 Tutorial +## 🔰 Tutorial Materials +### 1. Video Tutorial ✅ Watch [the portion about real-world Bayesian optimization and honegumi](https://youtu.be/IVaWl2tL06c?si=KQYNfAVw9f0AK-Ip&t=1874) from "A Gentle Introduction to Bayesian Optimization".
- +
-✅ Read through the homepage of -[Honegumi](https://honegumi.readthedocs.io/en/latest/), a Bayesian optimization -template generator. This tool will be used at various points during the course -to help with demonstrating advanced optimization topics. +### 2. Key Concepts + +#### Bayesian Optimization Fundamentals +- Surrogate model (Gaussian Process) +- Acquisition functions +- Exploration vs. exploitation trade-off +- Sequential decision making + +#### Practical Implementation +- Parameter space definition +- Constraint handling +- Experimental budget management +- Model validation techniques + +### 3. Tools and Framework +✅ Read through the [Honegumi documentation](https://honegumi.readthedocs.io/en/latest/): +- Framework overview +- Template generation +- Configuration options +- Best practices + +## 💻 Hands-on Tutorial :::{grid-item-card} Coding Tutorial Optimizing a 3D Printed Material for Strength Under Constraints @@ -28,28 +49,53 @@ Coding Tutorial ``` ::: -## 🚀 Quiz +### Step-by-Step Guide +1. Environment Setup + - Installing required packages + - Configuring development environment + +2. Problem Definition + - Defining parameter space + - Setting up constraints + - Creating objective function + +3. Optimization Implementation + - Using Honegumi templates + - Configuring AxClient + - Running experiments + +4. Results Analysis + - Model validation + - Feature importance analysis + - Visualization techniques + +## 📚 Additional Resources +### Documentation +- [Ax Platform Documentation](https://ax.dev/) +- [BoTorch Tutorials](https://botorch.org/tutorials/) +- [Gaussian Processes for Machine Learning](http://gaussianprocess.org/gpml/) + +### Research Papers +- ["A Tutorial on Bayesian Optimization"](https://arxiv.org/abs/1807.02811) +- ["Practical Bayesian Optimization of Machine Learning Algorithms"](https://papers.nips.cc/paper/2012/hash/05311655a15b75fab86956663e1819cd-Abstract.html) + +## 🚀 Quiz ::::{tab-set} :sync-group: category :::{tab-item} Sp/Su 2024 :sync: sp2024 - https://q.utoronto.ca/courses/370068/assignments/1327549 ::: - :::: ## 📄 Assignment - ::::{tab-set} :sync-group: category :::{tab-item} Sp/Su 2024 :sync: sp2024 - https://q.utoronto.ca/courses/370068/assignments/1327550 ::: - :::: diff --git a/docs/courses/data-science/2.2-multi-objective.md b/docs/courses/data-science/2.2-multi-objective.md index fd84cb9..a60e05a 100644 --- a/docs/courses/data-science/2.2-multi-objective.md +++ b/docs/courses/data-science/2.2-multi-objective.md @@ -1,36 +1,99 @@ # 🧩 2.2 Multi-objective Optimization -% ```{contents} -% :depth: 2 -% ``` +##�� Learning Objectives -## 🔰 Tutorial +After completing this tutorial, you will be able to: +1. Understand the difference between single- and multi-objective optimization +2. Apply Bayesian optimization to multi-objective problems +3. Interpret Pareto fronts and trade-offs +4. Implement constraints in multi-objective optimization -% TODO: Add side-by-side buttons! +## 🔰 Tutorial Materials ::::{grid} 1 2 2 2 -:::{grid-item-card} Concept Doc -Single- vs. Multi-objective Optimization +:::{grid-item-card} Concept Documentation +**Single- vs. Multi-objective Optimization** + +Key concepts covered: +- Trade-offs between competing objectives +- Pareto optimality +- Dominated vs non-dominated solutions +- Visualization of Pareto fronts + +++ ```{button-link} https://honegumi.readthedocs.io/en/latest/curriculum/concepts/sobo-vs-mobo/sobo-vs-mobo.html :color: info :expand: :click-parent: -Concept Doc +Read Concepts ``` ::: -:::{grid-item-card} Coding Tutorial -Multi Objective Optmization of Polymers for Strength and Biodegradability + +:::{grid-item-card} Hands-on Tutorial +**Multi-objective Optimization of Polymers** + +Practice exercises: +- Setting up optimization parameters +- Defining multiple objectives +- Adding constraints +- Analyzing Pareto optimal solutions +- Visualizing results + +++ ```{button-link} https://honegumi.readthedocs.io/en/latest/curriculum/tutorials/mobo/mobo.html :color: info :expand: :click-parent: -Coding Tutorial +Start Tutorial ``` ::: :::: +## 📚 Additional Resources + +### Key Concepts Review +1. **Pareto Optimality** + - Definition and importance + - How to identify Pareto optimal solutions + - Real-world applications + +2. **Trade-off Analysis** + - Understanding competing objectives + - Quantifying trade-offs + - Making informed decisions + +3. **Constraint Handling** + - Types of constraints + - Implementation strategies + - Impact on optimization + +### Honegumi Tools +1. **Optimization Setup** + ```python + from honegumi import MultiObjectiveOptimizer + + # Example setup + optimizer = MultiObjectiveOptimizer( + parameters=[...], + objectives=[...], + constraints=[...] + ) + ``` + +2. **Visualization Functions** + ```python + # Example visualization + optimizer.plot_pareto_front() + optimizer.plot_parallel_coordinates() + ``` + +3. **Results Analysis** + ```python + # Example analysis + optimal_points = optimizer.get_pareto_optimal() + trade_offs = optimizer.compute_trade_offs() + ``` + ## 🚀 Quiz ::::{tab-set} @@ -39,11 +102,37 @@ Coding Tutorial :::{tab-item} Sp/Su 2024 :sync: sp2024 -https://q.utoronto.ca/courses/370068/assignments/1327551 +Test your understanding: +- [Practice Quiz](https://q.utoronto.ca/courses/370068/assignments/1327551) +- Focus on concepts and implementation ::: :::: +## 📝 Pre-assignment Checklist + +Before starting the assignment, ensure you: + +1. **Understand Core Concepts** + - [ ] Pareto optimality + - [ ] Trade-off analysis + - [ ] Constraint implementation + +2. **Complete Tutorial** + - [ ] Read concept documentation + - [ ] Follow hands-on tutorial + - [ ] Test example code + +3. **Setup Environment** + - [ ] Install required packages + - [ ] Test Honegumi installation + - [ ] Verify visualization tools + +4. **Review Examples** + - [ ] Multi-objective optimization setup + - [ ] Constraint definition + - [ ] Results analysis + ## 📄 Assignment ::::{tab-set} @@ -52,7 +141,9 @@ https://q.utoronto.ca/courses/370068/assignments/1327551 :::{tab-item} Sp/Su 2024 :sync: sp2024 -https://q.utoronto.ca/courses/370068/assignments/1327552 +Ready for the challenge? +- [Assignment Link](https://q.utoronto.ca/courses/370068/assignments/1327552) +- Apply concepts to real-world epoxy optimization ::: :::: diff --git a/docs/courses/data-science/2.3-batch.md b/docs/courses/data-science/2.3-batch.md index 24a0a89..3ed8e00 100644 --- a/docs/courses/data-science/2.3-batch.md +++ b/docs/courses/data-science/2.3-batch.md @@ -1,13 +1,56 @@ # 🧩 2.3 Batch Optimization -% ```{contents} -% :depth: 2 -% ``` +## 📚 Learning Objectives -## 🔰 Tutorial +After completing this module, you should be able to: +- Understand the principles of batch optimization +- Implement batch-based Bayesian optimization +- Handle constraints in optimization problems +- Analyze optimization results effectively + +## 🔰 Prerequisites + +### Required Knowledge +- Basic understanding of Gaussian Processes +- Familiarity with Python programming +- Understanding of optimization concepts + +### Required Packages +```python +ax-platform>=0.3.0 # For Bayesian optimization +numpy>=1.20.0 # For numerical operations +pandas>=1.3.0 # For data manipulation +scipy>=1.7.0 # For scientific computing +``` + +## 📖 Core Concepts + +### Batch Optimization +Batch optimization allows for parallel evaluation of multiple design points, which is particularly useful when: +- Experiments are time-consuming +- Multiple samples can be processed simultaneously +- Resource constraints exist + +### Key Components +1. **Sampling Strategies** + - Initial Sobol sampling for exploration + - Batch Thompson sampling for exploitation + - Trade-off between exploration and exploitation + +2. **Constraint Handling** + - Device/system constraints + - Parameter bounds + - Feasibility checks + +3. **Performance Metrics** + - Optimization convergence + - Batch diversity + - Constraint satisfaction + +## 🔰 Tutorial Resources ::::{grid} 1 2 3 3 -:::{grid-item-card} Concept Doc \#1 +:::{grid-item-card} Concept Doc #1 Frequentist Vs. Fully Bayesian Gaussian Process Models +++ ```{button-link} https://honegumi.readthedocs.io/en/latest/curriculum/concepts/freq-vs-bayes/freq-vs-bayes.html @@ -17,7 +60,7 @@ Frequentist Vs. Fully Bayesian Gaussian Process Models Concept Doc ``` ::: -:::{grid-item-card} Concept Doc \#2 +:::{grid-item-card} Concept Doc #2 Single vs. Batch Optimization +++ ```{button-link} https://honegumi.readthedocs.io/en/latest/curriculum/concepts/batch/single-vs-batch.html @@ -39,6 +82,42 @@ Coding Tutorial ::: :::: +## 💡 Implementation Tips + +### Best Practices +1. **Parameter Setup** + - Validate parameter ranges + - Check for constraint feasibility + - Use appropriate data types + +2. **Optimization Process** + ```python + # Example structure + for batch in range(n_batches): + # Get next batch of parameters + parameters = get_next_batch() + + # Check constraints + validate_constraints(parameters) + + # Run experiments + results = run_batch_experiments(parameters) + + # Update model + update_optimization_model(results) + ``` + +3. **Results Analysis** + - Track convergence + - Monitor constraint violations + - Analyze batch diversity + +### Common Pitfalls +- Not handling constraints properly +- Insufficient initial exploration +- Poor parameter scaling +- Inadequate error handling + ## 🚀 Quiz ::::{tab-set} @@ -64,3 +143,33 @@ https://q.utoronto.ca/courses/370068/assignments/1327554 ::: :::: + +## 📚 Additional Resources + +### Reference Materials +- [Ax Platform Documentation](https://ax.dev/) +- [BoTorch Tutorials](https://botorch.org/tutorials/) +- [Bayesian Optimization Book](https://bayesoptbook.com/) + +### Related Topics +- Multi-objective optimization +- Constrained Bayesian optimization +- Sequential vs. batch sampling strategies + +## 🤔 FAQ + +1. **Q: Why use batch optimization?** + A: Batch optimization is efficient when experiments are time-consuming and multiple trials can be run in parallel. + +2. **Q: How to handle failed experiments?** + A: Implement proper error handling and consider using robust optimization strategies. + +3. **Q: What's the optimal batch size?** + A: It depends on your resources and the trade-off between exploration and exploitation. + +## 📝 Notes + +- Keep track of all experimental results +- Document any deviations from expected behavior +- Consider the computational cost of optimization +- Validate results against physical constraints diff --git a/docs/courses/data-science/2.5-multi-task.md b/docs/courses/data-science/2.5-multi-task.md index 0a68e65..7150412 100644 --- a/docs/courses/data-science/2.5-multi-task.md +++ b/docs/courses/data-science/2.5-multi-task.md @@ -1,11 +1,23 @@ -# 🧩 2.5 Multi-fidelity Optimization +# 🧩 2.5 Multi-task Optimization ```{contents} :depth: 2 ``` +## 📚 Introduction + +Multi-task Bayesian optimization (MTBO) is a powerful extension of traditional Bayesian optimization that allows us to simultaneously optimize related but distinct tasks. This approach is particularly valuable when we have multiple related optimization problems that share some underlying structure, but may have different optimal solutions. + ## 🔰 Tutorial +The following tutorial demonstrates how to implement multi-task Bayesian optimization using the Ax platform. We'll use two related objective functions: the standard Branin function and a shifted/inverted version of it, representing two different but related tasks. + +Key Components: +- Task encoding transforms +- Generation strategy setup +- Multi-task parameter configuration +- Alternating optimization between tasks + Run the following in an interpreter of your choice (`pip install ax-platform`). ```python @@ -116,6 +128,22 @@ print(f"Objective for task A: {objective_A}") print(f"Objective for task B: {objective_B}") ``` +## 💡 Key Concepts + +1. **Task Parameters**: Special parameters that identify which task is being optimized +2. **Transform Functions**: Help the model understand relationships between tasks +3. **Generation Strategy**: Combines initial exploration with model-based optimization +4. **Fixed Features**: Allow us to specify which task to optimize in each iteration + +## 🔑 Best Practices + +When implementing multi-task optimization: +1. Ensure tasks are genuinely related +2. Balance experiments between tasks +3. Consider using task-specific constraints +4. Monitor performance differences between tasks +5. Validate transfer learning benefits + ::::{grid} 1 2 2 2 :::{grid-item-card} Concept Doc Multitask Bayesian Optimization @@ -164,3 +192,9 @@ https://q.utoronto.ca/courses/370068/assignments/1327558 ::: :::: + +## 📌 Additional Resources + +- [BoTorch Multi-task GP Documentation](https://botorch.org/tutorials/multi_task_gp) +- [Ax Platform Examples](https://ax.dev/tutorials/multitask.html) +- [Multi-task Bayesian Optimization Research Paper](https://proceedings.neurips.cc/paper/2011/file/f3f1b7fc5a8779a9e618e1f23a7b7860-Paper.pdf) diff --git a/docs/courses/hello-world/1.5-data-logging.md b/docs/courses/hello-world/1.5-data-logging.md index 26d1b99..ad479dd 100644 --- a/docs/courses/hello-world/1.5-data-logging.md +++ b/docs/courses/hello-world/1.5-data-logging.md @@ -1,7 +1,4 @@ -# 🧩 1.5 Logging Data - -% ```{rubric} 🧩 1.5 Logging Data -% ``` +# �� 1.5 Logging Data ```{contents} :depth: 3 @@ -10,10 +7,10 @@ ## 🔰 Tutorial ```{warning} -As a temporary workaround to the issue described in https://github.com/orgs/micropython/discussions/15112, add [`urequests_2.py`](https://github.com/AccelerationConsortium/ac-microcourses/blob/main/docs/courses/hello-world/urequests_2.py) [[permalink](https://github.com/AccelerationConsortium/ac-microcourses/blob/e5541ce3ec307a8e5e0f2b20f000c03f040e1f56/docs/courses/hello-world/urequests_2.py)] to your microcontroller, and change `import urequests` to `import urequests_2 as urequests` in the code below. See https://github.com/orgs/micropython/discussions/15112 and https://github.com/micropython/micropython-lib/pull/861 for ongoing updates. The corresponding assignment will also be affected, but this can be addressed using the same workaround. +As a temporary workaround to the issue described in https://github.com/orgs/micropython/discussions/15112, add [`urequests_2.py`](https://github.com/AccelerationConsortium/ac-microcourses/blob/main/docs/courses/hello-world/urequests_2.py) to your microcontroller, and change `import urequests` to `import urequests_2 as urequests` in the code below. ``` -In this tutorial, you will learn how to upload data to a MongoDB database directly from a microcontroller and read data from a database using Python. +In this tutorial, you will learn how to upload data to a MongoDB database from a microcontroller and read data from a database using Python with PyMongo. @@ -27,173 +24,66 @@ Chemistry and materials research data is precious. By making your data **F**inda *Defining FAIR Data in materials science. Reproduced from https://doi.org/10.1557/s43577-023-00498-4* -### # Updating Pico W to Use AWS Lambda for MongoDB Interaction - -For storing our data, we will be using [MongoDB](https://www.mongodb.com/) as before, but instead of using MongoDB's Data API directly, we'll be using [AWS Lambda](https://aws.amazon.com/lambda/) as an intermediary. AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. It's an excellent choice for creating lightweight APIs and handling intermittent requests, which is perfect for our IoT application. - -By using AWS Lambda in conjunction with [Amazon API Gateway](https://aws.amazon.com/api-gateway/), we can create a secure and scalable API that acts as a bridge between our Pico W and MongoDB. This approach offers several advantages: - -1. Enhanced security: We can implement more robust authentication and authorization mechanisms. -2. Flexibility: We can easily add data processing or validation logic before writing to MongoDB. -3. Cost-effective: Lambda's pay-per-use model can be more economical for intermittent data logging. - -For this tutorial, we've set up a Lambda function that connects to our MongoDB database. The function is triggered via an API Gateway endpoint, which the Pico W will call to log data. To maintain security, the API Gateway is configured with an API key, and the Lambda function has the necessary permissions to write to MongoDB. - -✅ Copy the following code into a new file on the microcontroller called `write_aws_lambda.py` and run the file. Note that you will need [`netman.py`](https://github.com/sparks-baird/self-driving-lab-demo/blob/main/src/public_mqtt_sdl_demo/lib/netman.py) [[permalink](https://github.com/sparks-baird/self-driving-lab-demo/blob/0ff0adec3e997c096990de594844d73a9ce18fd6/src/public_mqtt_sdl_demo/lib/netman.py)] and a file named `my_secrets.py` with your WiFi credentials, course ID, and the AWS API Gateway URL and API key. - -This guide explains how to modify the Pico W code to use AWS Lambda as a data API interface for MongoDB, instead of using MongoDB's Data API directly. - -## Setting up AWS Lambda and API Gateway - -## Required Reading Materials - -- [AWS Lambda Documentation](https://docs.aws.amazon.com/lambda/) -- [AWS API Gateway Documentation](https://docs.aws.amazon.com/apigateway/) -- [Raspberry Pi Pico W Documentation](https://www.raspberrypi.com/documentation/microcontrollers/raspberry-pi-pico.html) - -## Required Python Files - -1. `main.py`: Contains the modified code above -2. `netman.py`: Module for WiFi connection (remains unchanged) -3. `my_secrets.py`: Contains WiFi credentials and COURSE_ID - -## Setting up AWS Lambda and API Gateway for MongoDB Integration - -Follow these steps to set up AWS Lambda and API Gateway to interact with MongoDB: - -1. Create an AWS Lambda function: - - Go to AWS Lambda console and click "Create function". - - Choose "Author from scratch" and give your function a name. - - Select Python as the runtime. - - Create a new role with basic Lambda permissions. - -2. Set up the Lambda function: - - In the function code, paste the Python code provided in the "Lambda Function Code" section below. - - Add the `pymongo` layer to your Lambda function for MongoDB connectivity. - - In the function configuration, add an environment variable for your MongoDB connection string. - -3. Create an API Gateway: - - Go to API Gateway console and create a new REST API. - - Create a new resource and add a POST method. - - Set the integration type to "Lambda Function" and select your created function. - -4. Set up API Key: - - In API Gateway, create a new API key. - - Associate this key with your API and deployed stage. +### Data Logging Architecture -5. Deploy the API: - - Deploy your API to a new stage (e.g., "prod"). - - Note the invoke URL - this will be your `AWS_API_GATEWAY_URL`. - - Note the API key - this will be your `AWS_API_KEY`. +For storing our data, we will be using [MongoDB](https://www.mongodb.com/). The architecture consists of two main components: -6. Update your Pico W code: - - Set up the AWS API Gateway URL and API key in your `my_secrets.py` file. - - Prepare the document data to be sent in your main script. - - Send a POST request to the AWS API Gateway. - - Handle the response and implement retry logic if necessary. +1. **Microcontroller Data Upload**: The microcontroller (Pico W) collects data and sends it to the orchestrator +2. **Orchestrator Data Management**: The orchestrator, running a full Python environment, uses PyMongo to interact with MongoDB for data storage and retrieval -7. Ensure security: - - Verify that the Lambda function has appropriate permissions to access MongoDB. - - Keep your API key secure and do not share it publicly. +This setup allows us to leverage the full capabilities of PyMongo in the orchestrator while keeping the microcontroller code simple and focused on data collection. -By following these steps, you'll have set up a serverless API that allows your Pico W to securely write data to MongoDB through AWS Lambda. -## Code Modifications +### Setting Up the Data Pipeline -Modify the `main.py` file as follows: +✅ Copy the following code into a new file on the microcontroller called `send_data.py`. Note that you will need [`netman.py`](https://github.com/sparks-baird/self-driving-lab-demo/blob/main/src/public_mqtt_sdl_demo/lib/netman.py) and a file named `my_secrets.py` with your WiFi credentials and orchestrator endpoint details. ```python from netman import connectWiFi -import urequests -from my_secrets import SSID, PASSWORD, COURSE_ID, AWS_API_GATEWAY_URL, AWS_API_KEY +import urequests_2 as urequests # Using urequests_2 as per warning +from my_secrets import SSID, PASSWORD, ORCHESTRATOR_URL +# Connect to WiFi connectWiFi(SSID, PASSWORD, country="US") -headers = {"x-api-key": AWS_API_KEY} -document = {"course_id": COURSE_ID} - -print(f"Sending document to AWS Lambda") - -num_retries = 3 -for _ in range(num_retries): - response = urequests.post(AWS_API_GATEWAY_URL, headers=headers, json=document) - txt = str(response.text) - status_code = response.status_code - - if status_code != 200: - print("Retrying in 5 seconds...") - time.sleep(5) - - print(f"Response: ({status_code}), msg = {txt}") - - response.close() - - if status_code == 200: - print("Added Successfully") - break - - print("Retrying...") +# Prepare sample data +data = { + "temperature": 25.5, + "humidity": 60, + "timestamp": "2024-03-20T10:00:00" +} + +# Send data to orchestrator +response = urequests.post(ORCHESTRATOR_URL, json=data) +print(f"Response status: {response.status_code}") +response.close() ``` +### Working with MongoDB in the Orchestrator -### Example of `my_secrets.py`: +The orchestrator uses PyMongo to interact with MongoDB. Here's a basic example of how to connect and write data: ```python -SSID = "Your_WiFi_SSID" -PASSWORD = "Your_WiFi_Password" -COURSE_ID = "Your_Course_ID" -AWS_API_GATEWAY_URL = "Your_API_Gateway_Invoke_URL" -AWS_API_KEY = "Your_API_Key" +from pymongo import MongoClient + +# Connect to MongoDB +client = MongoClient('mongodb://localhost:27017/') +db = client['your_database'] +collection = db['your_collection'] + +# Insert data +data = { + "temperature": 25.5, + "humidity": 60, + "timestamp": "2024-03-20T10:00:00" +} +result = collection.insert_one(data) ``` -## Example Lambda Function (Python) -```python -import json -import pymongo -import os - - -def lambda_handler(event, context): - try: - client = pymongo.MongoClient(os.environ["MONGODB_URI"]) - db = client["your_database_name"] - collection = db["your_collection_name"] - - body = json.loads(event["body"]) - result = collection.insert_one(body) - - return { - "statusCode": 200, - "body": json.dumps( - { - "message": "Document inserted successfully", - "id": str(result.inserted_id), - } - ), - } - except Exception as e: - return { - "statusCode": 500, - "body": json.dumps( - {"message": "Error inserting document", "error": str(e)} - ), - } -``` - -The output should look something like the following (MAC address and IP address redacted, and your insertedId will be different): - -> ```python -> MAC address: *** -> connected -> ip = *** -> sending document to AWS Lambda -> Response: (200), msg = {"message": "Document inserted successfully"} -> Added Successfully -> ``` - -### Reading Data via AWS Lambda - -✅ Run the code from [the companion notebook](./1.5.1-aws-lambda-read.ipynb) to read data from MongoDB through AWS Lambda. +For more detailed examples of reading and writing data using PyMongo, including querying, updating, and managing collections, please refer to our [PyMongo tutorial notebook](./1.5.1-pymongo.html). +```{tip} +When working with MongoDB in Python, it's recommended to use connection pooling and proper error handling. See the [PyMongo documentation](https://pymongo.readthedocs.io/en/stable/examples/index.html) for best practices. +``` ### Reading Material @@ -207,17 +97,13 @@ The output should look something like the following (MAC address and IP address - FAIR data principles [website](https://www.go-fair.org/fair-principles/) and [manuscript](https://doi.org/10.1038/sdata.2016.18) - [MongoDB: What is NoSQL?](https://www.mongodb.com/nosql-explained) - [MongoDB Getting Started Interactive Tutorial](https://www.mongodb.com/docs/manual/tutorial/getting-started/) -- [Connecting a Raspberry Pi Pico to MongoDB Atlas](https://medium.com/@johnlpage/introduction-to-microcontrollers-and-the-pi-pico-w-f7a2d9ad1394) +- [PyMongo Documentation](https://pymongo.readthedocs.io/) - [@sgbaird's list of materials databases](https://github.com/stars/sgbaird/lists/materials-databases) -- Misc. [`awesome-materials-informatics`](https://github.com/tilde-lab/awesome-materials-informatics) (search for "database"), [`Materials-Databases` (no longer maintained)](https://github.com/blaiszik/Materials-Databases), [`awesome-chemistry-datasets`](https://github.com/kjappelbaum/awesome-chemistry-datasets) +- Misc. [`awesome-materials-informatics`](https://github.com/tilde-lab/awesome-materials-informatics) - [`awesome-self-driving-labs` research data management section](https://github.com/AccelerationConsortium/awesome-self-driving-labs#research-data-management) -As a side note, MongoDB has a "serverless" option (i.e., pay only for what you use) that exceeds the free-tier limits and is more flexible than the shared and dedicated clusters, which may seem appealing at first. However, [costs will escalate quickly if the database is not optimized](https://www.mongodb.com/developer/products/atlas/serverless-instances-billing-optimize-bill-indexing/) (e.g., the database is not indexed). If you decide to go with MongoDB Atlas for a project and need more than the 512 MB of free-tier storage, we recommend first considering the shared (e.g., M2, M5) and dedicated options, hosting your own MongoDB instance, or looking to other ecosystems. Looking to other ecosystems could be either a full replacement or a supplement. - ```{tip} -Upload larger files to [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) and store the URI in MongoDB. - -For example, let's say that one step in your workflow involves capturing images, where the total storage required is on the order of GBs instead of MBs. Instead of saving your images directly to MongoDB (e.g., using [GridFS or the `BinData` type](https://www.mongodb.com/docs/manual/core/gridfs/)), you can upload your image to [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) or similar and add the uniform resource identifier (URI) to the corresponding MongoDB document along with other data like sample name, image acquisition parameters, material composition, and processing conditions. The URI can then later be used to [programatically access the file from Amazon S3](https://chat.openai.com/share/4ec06c80-8915-4b21-9b3c-4e7a9abb186a). +For larger files like images, consider using [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) and storing only the URI in MongoDB. This approach is more efficient for managing large binary data while maintaining the relationship between your experimental data and associated files. ``` ## 🚀 Quiz diff --git a/docs/courses/hello-world/1.6-connecting-the-pieces.md b/docs/courses/hello-world/1.6-connecting-the-pieces.md index 83c915b..de48712 100644 --- a/docs/courses/hello-world/1.6-connecting-the-pieces.md +++ b/docs/courses/hello-world/1.6-connecting-the-pieces.md @@ -1,8 +1,5 @@ # 🧩 1.6 Connecting the Pieces -% ```{rubric} 🧩 1.6 Connecting the Pieces -% ``` - ```{contents} :depth: 2 ``` @@ -10,10 +7,7 @@ ## 🔰 Tutorial ```{warning} -As a temporary workaround to the issue described in https://github.com/orgs/micropython/discussions/15112, add [`urequests_2.py`](https://github.com/AccelerationConsortium/ac-microcourses/blob/main/docs/courses/hello-world/urequests_2.py) [[permalink](https://github.com/AccelerationConsortium/ac-microcourses/blob/e5541ce3ec307a8e5e0f2b20f000c03f040e1f56/docs/courses/hello-world/urequests_2.py)] to your microcontroller, and change `import urequests` to `import urequests_2 as urequests` in the code below. See https://github.com/orgs/micropython/discussions/15112 and https://github.com/micropython/micropython-lib/pull/861 for ongoing updates. The corresponding assignment will also be affected, but this can be addressed using the same workaround. -``` - -```{include} ./mqtt-warning.md +As a temporary workaround to the issue described in https://github.com/orgs/micropython/discussions/15112, add [`urequests_2.py`](https://github.com/AccelerationConsortium/ac-microcourses/blob/main/docs/courses/hello-world/urequests_2.py) to your microcontroller, and change `import urequests` to `import urequests_2 as urequests` in the code below. ``` You've made it to the final module of this microcourse! Well done 🥳. In this module, you will connect the pieces from previous modules into the "Hello World" of self-driving labs, from microcontroller programming and Bayesian optimization to hardware-software communication and data logging. @@ -32,23 +26,17 @@ You've made it to the final module of this microcourse! Well done 🥳. In this 4. [Hardware-software communication](./1.4-hardware-software-communication) 5. [Data logging](1.5-data-logging) -✅ Review the knowledge checks via the main course website +### Data Logging with PyMongo -✅ Review your assignment repositories. For quick access, see [your ACC-HelloWorld GitHub organization repositories](https://github.com/orgs/ACC-HelloWorld/repositories). +In the previous module, you learned how to log data using PyMongo directly within orchestrators. For detailed instructions, refer to the [PyMongo tutorial notebook](https://ac-microcourses.readthedocs.io/en/latest/courses/hello-world/1.5.1-pymongo.html). - +This approach allows you to efficiently manage data from your microcontroller using a full Python environment, leveraging the capabilities of PyMongo for database interactions. - +✅ Review your assignment repositories. For quick access, see [your ACC-HelloWorld GitHub organization repositories](https://github.com/orgs/ACC-HelloWorld/repositories). ## 🚀 Quiz