Instructors:
Cody Hennesy (UCB), Jose Niño Muriel (UCSB), Jamie Jamison (UCLA), Tim Dennis (UCLA), Seth Erickson (UCSB), Ryan Horne (UCLA), Derek Devnich (UCM), Reid Otsuji (UCSD)
Helpers:
Dave George (UCLA), Kristi Liu (UCSB), Geno Sanchez (UCLA), Renee Hui Xin Ng (UCSD), Hannah Sutherland (UCLA), Geoffery Boushey (UCSF), Kat Koziar (Fresno State), David Palmquist (Fullerton)
General Information
Registration: Please register in Zoom before attending the workshop.
The Carpentries project comprises the Software Carpentry, Data Carpentry, and
Library Carpentry communities of Instructors, Trainers, Maintainers,
helpers, and supporters who share a mission to teach foundational computational and data science
skills to researchers.
Want to learn more and stay engaged with The Carpentries? Carpentries Clippings is The Carpentries' biweekly newsletter, where we share community news, community job postings, and more.
Sign up to receive future editions and read our full archive: https://carpentries.org/newsletter/
Library Carpentry
is made by people working in library- and information-related roles to help you:
automate repetitive, boring, error-prone tasks
create, maintain and analyze sustainable and reusable data
Who:
The course is for people working in library- and information-related roles.
You don't need to have any previous knowledge of the tools that
will be presented at the workshop.
Where: This training will take place online.
The instructors will provide you with the information you will need to connect to this meeting.
Requirements:
Participants must have access to a computer with a
Mac, Linux, or Windows operating system (not a tablet, Chromebook, etc.) that they have administrative privileges on.
They should have a few specific software packages installed (listed below).
Accessibility:
We are committed to making this workshop
accessible to everybody.
We are dedicated to providing a positive and accessible learning environment for all.
We do not require participants to provide documentation of disabilities or disclose any unnecessary personal information.
However, we do want to help create an inclusive, accessible experience for all participants.
We encourage you to share any information that would be helpful to make your Carpentries experience accessible.
To request accessibility support for this workshop, please fill out the
accessibility support request form.
If you have questions or need assistance with the accessibility support form please email us.
Glosario is a multilingual glossary
for computing and data science terms. The glossary helps
learners attend workshops and use our lessons to make sense of computational and programming jargon written in English by offering it
in their native language. Translating data science terms also provides a teaching tool for Carpentries Instructors to reduce barriers
for their learners.
Workshop Recordings:
Carpentries workshops are designed to be interactive rather than lecture-based, with lessons that build upon one another.
To foster a positive online learning environment, we strongly recommend that participants join in real time.
As a result, workshop recordings are not recommended and may not be available to learners.
Roles:
To learn more about the roles at the workshop (who will be doing what),
refer to our Workshop FAQ.
Code of Conduct
Everyone who participates in Carpentries activities is required to conform to the Code of Conduct. This document also outlines how to report an incident if needed.
Collaborative Notes
We will use this collaborative document for chatting, taking notes, and sharing URLs and bits of code.
Surveys
Please be sure to complete these surveys before and after the workshop.
Mid-Workshop Break: No sessions Thursday, May 14 – Sunday, May 17.
Day 4: Monday, May 18, OpenRefine
09:00
Introduction to OpenRefine
09:30
Importing & Exploring Data
10:15
Faceting and Filtering
10:30
Break
10:45
Clustering & Working with Columns
11:30
Transformations (GREL) & Exporting
12:00
Wrap-up
Day 5: Tuesday, May 19, Python (Part 1)
09:00
Python: Variables & Types
09:45
Lists & Built-in Functions
10:30
Break
10:45
Libraries & Introduction to Pandas
11:30
For Loops & Looping Over Datasets
12:00
Wrap-up
Day 6: Wednesday, May 20, Python (Part 2)
09:00
Advanced Pandas & Filtering Data
10:00
Conditionals & Writing Functions
10:30
Break
10:45
Data Visualization with Python
11:30
Post-workshop Survey & Wrap-up
12:00
End of Workshop
Setup
To participate in a
Library Carpentry
workshop,
you will need access to software as described below.
In addition, you will need an up-to-date web browser.
If you haven't used Zoom before, go to the
official website
to download and install the Zoom client for your computer.
Set up your workspace
Like other Carpentries workshops,
you will be learning by "coding along" with the Instructors.
To do this, you will need to have both the window for the tool
you will be learning about (a terminal, RStudio, your web browser, etc..)
and the window for the Zoom video conference client open.
In order to see both at once,
we recommend using one of the following set up options:
Two monitors: If you have two monitors,
plan to have the tool you are learning up on one monitor and
the video conferencing software on the other.
Two devices: If you don't have two monitors,
do you have another device (tablet, smartphone) with a medium to large
sized screen? If so, try using the smaller device as your video
conference connection and your larger device (laptop or desktop)
to follow along with the tool you will be learning about.
Divide your screen: If you only have one device
and one screen, practice having two windows
(the video conference program and one of the tools you will be using
at the workshop) open together.
How can you best fit both on your screen?
Will it work better for you to toggle between them
using a keyboard shortcut?
Try it out in advance to decide what will work best for you.
This blog post includes detailed information on how to set up your screen to follow along during the workshop.
Spreadsheet Software
To interact with spreadsheets in this lesson, we will use
Microsoft Excel. If you don't have access to Excel, we recommend using Google Sheets. Google Sheets is free with a (free) Google or Gmail account. You can sign in at Google Workspace.
There are other free spreadsheet tools that you can use, though we may not have time to answer questions during the workshop about differences between all of these:
LibreOffice,
Gnumeric,
OpenOffice.org.
Commands may differ a bit between programs, but general ideas for thinking about spreadsheets is the same.
The Bash Shell
Bash is a command-line tool that lets you work with files and folders by typing commands.
For this lesson, you will need a Unix-like shell environment and the lesson data.
Please complete the setup below before the workshop. If you get stuck, stop and we will help you during the workshop.
Unzip or extract the file. You should end up with a folder on your Desktop called shell-lesson.
Test your setup
Open your shell and run:
ls
pwd
If you can type commands and see output, your shell is ready. During the workshop, we will use the shell-lesson folder on your Desktop.
If something goes wrong
If you are on Windows, make sure you are opening Git Bash, not Command Prompt or PowerShell.
If you cannot find the data, check that shell-lesson is on your Desktop.
If you get stuck, stop and we will help you during the workshop.
OpenRefine
For this lesson you will use OpenRefine, a tool for cleaning and organizing data.
It runs on your computer and opens in your web browser (no internet connection needed after launch).
Please install OpenRefine and confirm it opens before the workshop.
Save it somewhere easy to find, such as your Desktop or Downloads folder.
During the workshop, we will open this file in OpenRefine.
Test your setup
OpenRefine should open in your browser and show a page with options like
"Create Project" and "Open Project".
If something goes wrong
If nothing opens, go to http://127.0.0.1:3333 in your browser.
If you see a security warning on Mac, use Control-click → Open.
If you get stuck, stop and we will help you during the workshop.
Git
Git is a version control system that helps you track changes to files and collaborate with others.
In this workshop, we will also use GitHub to share code.
Please make sure Git is installed and that you can sign in to GitHub before the workshop.
Create a GitHub account
You will need a free GitHub account for this lesson.
If you already have one, please make sure you can sign in before the workshop.
Git is installed as part of Git Bash. If you completed the Shell setup, you should already have Git.
Open Git Bash.
Run:
git --version
You should see a version number, such as git version 2.x.x.
Open the Terminal application.
Run:
git --version
If Git is not installed, you may be prompted to install the Command Line Tools.
Follow the prompts to complete installation.
After installation, run git --version again to confirm it is working.
Open a terminal.
Run:
git --version
If Git is not installed, use your package manager. For example:
sudo apt-get install git
Then run git --version again to confirm installation.
Test your setup
Run git --version. You should see a version number.
Also make sure you can sign in to your GitHub account.
If something goes wrong
If git --version is not found, Git is not installed yet.
Try restarting your terminal after installing Git.
If you cannot sign in to GitHub, reset your password or finish email verification before the workshop.
If you get stuck, stop and we will help you during the workshop.
Python
We will use Python to write and run code, and
JupyterLab as our workspace.
JupyterLab opens in a web browser but runs on your computer.
We recommend Miniforge for installation.
Miniforge includes Python. You do not need to download Python separately.
It also includes conda, a tool for installing software and keeping
project software in separate environments. In the steps below, you will create
a workshop environment called carpentries. This step installs
JupyterLab and the Python packages used in the lesson.
Complete setup before the workshop if you can. If a step fails, stop there and
contact the workshop organizers. Save the error message or take a screenshot
so we can help. If your computer requires IT approval to install software,
request that approval before the workshop.
Before you begin
Use an internet connection for the downloads and installation.
If you already use Anaconda, Miniconda, or Miniforge, ask the organizers whether you can use your existing installation before installing another copy.
Choose your operating system below and follow its steps in order. Use the Windows instructions unless you already work in Windows Subsystem for Linux (WSL).
We will use a folder called lc-python inside your home folder. This is a different location from the Desktop folder mentioned in the lesson's general setup page.
A terminal is a window where you type commands.
On Windows, use Miniforge Prompt; on macOS or Linux, use
Terminal. Copy one command line at a time and press
Enter (or Return on a Mac).
Wait until the prompt returns before entering the next command.
The prompt is the text or symbol beside the cursor when the terminal is ready
for another command. Do not copy prompt symbols such as $ or >.
Visit the Miniforge download page and choose Windows x86_64. These steps are for a Windows PC with an Intel or AMD processor. If your PC uses an ARM processor, contact the organizers for help choosing an installation route.
Open the downloaded .exe file. Read and accept the license, choose Just Me, and keep the default installation folder.
Keep Create start menu shortcuts selected. Leave Add Miniforge3 to my PATH environment variable unchecked. You will use Miniforge Prompt, so adding Miniforge to PATH is not needed.
Finish the installation. Open the Windows Start menu, search for Miniforge Prompt, and open it.
Check that conda is available:
conda --version
You should see conda followed by a version number. Your number may differ from examples you see elsewhere.
2. Prepare your workshop files
In File Explorer, click the address bar, type %USERPROFILE%, and press Enter. This opens your home folder.
Create a folder there called lc-python. If it already exists, use that folder.
Right-click carpentries_environment.yml and choose Save link as (the wording varies by browser). Save it inside lc-python. If the save dialog offers a file type, choose All files. Keep the filename exactly carpentries_environment.yml, with no extra .txt extension.
Right-click data.zip and choose Extract All. Set the destination to your lc-python folder. After extraction, that folder should contain a data folder with CSV files inside it. If extraction creates an extra enclosing folder, move the folder containing the CSV files into lc-python and name it data.
Your lc-python folder should now contain carpentries_environment.yml and a data folder. It may also contain data.zip, which you can keep.
3. Create the workshop environment
Return to Miniforge Prompt. Run these commands one line at a time:
cd /d "%USERPROFILE%\lc-python"
conda env create -f carpentries_environment.yml
This downloads and installs the workshop software. It may take several minutes and display many lines of text. Wait for the prompt to return. If you see an error, stop and use the troubleshooting section below.
4. Open JupyterLab
conda activate carpentries
jupyter lab
JupyterLab should open in your browser. Keep Miniforge Prompt open while you use it. Continue to Check that setup works below the tabs.
5. Open it again on workshop day
Open Miniforge Prompt and run:
cd /d "%USERPROFILE%\lc-python"
conda activate carpentries
jupyter lab
You only need to create the environment once.
macOS
1. Install Miniforge
Open the Apple menu → About This Mac. If you see an Apple M-series chip, choose arm64 (Apple Silicon) on the Miniforge download page. If you see an Intel processor, choose x86_64 (Intel).
Save the installer in your Downloads folder. These instructions use the file ending in .sh, which you run in Terminal. If you downloaded a .pkg instead, use that installer's instructions or ask the organizers for help.
Open Terminal: press Command+Space, type Terminal, and press Return.
Run one of the commands below, matching the installer you downloaded.
Apple Silicon:
bash ~/Downloads/Miniforge3-MacOSX-arm64.sh
Intel:
bash ~/Downloads/Miniforge3-MacOSX-x86_64.sh
Follow the installer prompts:
Press Return to begin. Read the license, using Space to advance through it. Type yes and press Return to accept.
Press Return to accept the default installation location, ~/miniforge3. The ~ symbol means your home folder.
Type yes when asked whether to initialize conda. This sets up your terminal to find conda.
When installation finishes, close the Terminal window and open a new one.
Check that conda is available:
conda --version
You should see conda followed by a version number.
2. Prepare your workshop files
In Finder, choose Go → Home. Create a folder called lc-python. If it already exists, use that folder.
Control-click carpentries_environment.yml and choose Download Linked File As or Save Link As (the wording varies by browser). Save it inside lc-python. Keep the filename exactly carpentries_environment.yml, with no extra .txt extension.
Double-click data.zip to extract it. If your browser already extracted it in Downloads, move the resulting data folder into lc-python. Check that data contains CSV files.
Your lc-python folder should now contain carpentries_environment.yml and a data folder. It may also contain data.zip, which you can keep.
3. Create the workshop environment
In Terminal, run these commands one line at a time:
cd ~/lc-python
conda env create -f carpentries_environment.yml
This downloads and installs the workshop software. It may take several minutes and display many lines of text. Wait for the prompt to return. If you see an error, stop and use the troubleshooting section below.
4. Open JupyterLab
conda activate carpentries
jupyter lab
JupyterLab should open in your browser. Keep Terminal open while you use it. Continue to Check that setup works below the tabs.
5. Open it again on workshop day
Open Terminal and run:
cd ~/lc-python
conda activate carpentries
jupyter lab
You only need to create the environment once.
Linux
1. Install Miniforge
Open your terminal application. Run the following command to check your computer's processor architecture:
uname -m
Visit the Miniforge download page. Choose Linux x86_64 if the command returned x86_64, or Linux aarch64 if it returned aarch64. If you see another result, ask the organizers for help.
Save the .sh installer in your Downloads folder. Run one of the commands below, matching the installer you downloaded.
x86_64:
bash ~/Downloads/Miniforge3-Linux-x86_64.sh
aarch64:
bash ~/Downloads/Miniforge3-Linux-aarch64.sh
Follow the installer prompts:
Press Enter to begin. Read the license, using Space to advance through it. Type yes and press Enter to accept.
Press Enter to accept the default installation location, ~/miniforge3. The ~ symbol means your home folder.
Type yes when asked whether to initialize conda.
When installation finishes, close the terminal window and open a new one.
Check that conda is available:
conda --version
You should see conda followed by a version number.
2. Prepare your workshop files
In your file manager, open your Home folder. Create a folder called lc-python. If it already exists, use that folder.
Right-click carpentries_environment.yml and choose Save Link As (the wording varies by browser). Save it inside lc-python. Keep the filename exactly carpentries_environment.yml, with no extra .txt extension.
Use your archive manager to extract data.zip into lc-python. Check that the resulting data folder contains CSV files. If extraction creates an extra enclosing folder, move the folder containing the CSV files into lc-python and name it data.
Your lc-python folder should now contain carpentries_environment.yml and a data folder. It may also contain data.zip, which you can keep.
3. Create the workshop environment
In your terminal, run these commands one line at a time:
cd ~/lc-python
conda env create -f carpentries_environment.yml
This downloads and installs the workshop software. It may take several minutes and display many lines of text. Wait for the prompt to return. If you see an error, stop and use the troubleshooting section below.
4. Open JupyterLab
conda activate carpentries
jupyter lab
JupyterLab should open in your browser. Keep the terminal open while you use it. Continue to Check that setup works below the tabs.
5. Open it again on workshop day
Open your terminal and run:
cd ~/lc-python
conda activate carpentries
jupyter lab
You only need to create the environment once.
Windows Subsystem for Linux (WSL)
Use these steps only if WSL is already installed and you normally work in it. You do not need to install WSL for this workshop. Otherwise, use the Windows tab.
1. Install Miniforge inside WSL
Open your WSL Linux terminal, such as Ubuntu, from the Windows Start menu. These commands belong in that Linux terminal.
Run the commands below one line at a time. They download the Linux installer into your Linux home folder and then run it. If curl is not available, stop and contact the organizers.
Press Enter to begin. Read the license, using Space to advance through it. Type yes and press Enter to accept.
Press Enter to accept the default installation location, ~/miniforge3. The ~ symbol means your Linux home folder.
Type yes when asked whether to initialize conda.
When installation finishes, close the WSL terminal window and open a new one.
Check that conda is available:
conda --version
You should see conda followed by a version number.
2. Prepare your workshop files
Run these commands one line at a time in WSL. They create your workshop folder and download the files directly into it, so you do not need to move files from Windows Downloads.
The final command extracts the ZIP file into the current folder. Check the contents:
ls
ls data
The first command should list carpentries_environment.yml and data, along with data.zip. The second should list CSV files. If your folders differ, contact the organizers.
3. Create the workshop environment
conda env create -f carpentries_environment.yml
This downloads and installs the workshop software. It may take several minutes and display many lines of text. Wait for the prompt to return. If you see an error, stop and use the troubleshooting section below.
Keep the WSL terminal open. Copy the complete address beginning with http://localhost: from the terminal into your Windows browser, including the part containing token=. This token grants access to your JupyterLab session; do not share it. Continue to Check that setup works below the tabs.
5. Open it again on workshop day
Open your WSL terminal and run:
cd ~/lc-python
conda activate carpentries
jupyter lab --no-browser
Use the new browser address printed in the terminal. You only need to create the environment once.
Check that setup works
In JupyterLab, find the file browser on the left. You should see your data folder. Double-click it and check that CSV files are inside, then return to lc-python using the folder path above the file list.
Choose File → New Launcher. Under Notebook, click Python 3 (the label may also include ipykernel).
Click the first empty cell in the notebook and type the code below. Press Shift+Enter to run it.
import pandas as pd
print("Setup works!")
If you see Setup works! below the cell with no error, and you can see the lesson CSV files, your setup is ready.
Use the data downloaded here. These files were modified for teaching and may differ from versions linked from data.gov.
When you finish
Save any notebook work with Ctrl+S (Windows or Linux) or Command+S (Mac). To stop JupyterLab, return to the terminal window where you started it and press Ctrl+C. If asked to shut down the server, type y and press Enter. You can then close the browser tab and terminal.
Troubleshooting
Windows blocks the installer
Check that you downloaded the installer from the official Miniforge download page linked above. Some downloaded files have an Unblock checkbox under right-click → Properties → General. If that option is present and your organization's rules permit it, select it and try again. This does not resolve every type of security block. If Windows says your organization or administrator has blocked the app, contact IT or the organizers for help.
The installer file cannot be found
Check where your browser saved the installer and whether its name matches the command. The macOS and Linux commands assume it is in your home folder's Downloads folder with its original filename. If your browser added a suffix such as (1), the name will differ. Ask the organizers for help rather than repeatedly downloading the installer.
Windows says conda is not recognized
Open Miniforge Prompt from the Start menu and try conda --version there. The workshop commands are written for Miniforge Prompt.
macOS, Linux, or WSL says conda: command not found
First close the terminal window and open a new one. If the error continues and you accepted the default installation location, run the command for your shell below.
macOS using the default zsh shell:
~/miniforge3/bin/conda init zsh
Linux or WSL using bash:
~/miniforge3/bin/conda init bash
Then close the terminal window, open a new one, and try conda --version again. If you chose another installation folder, use another shell, or still see an error, contact the organizers.
The environment file cannot be found or read
Make sure you ran the cd command for your operating system and saved carpentries_environment.yml inside lc-python. Check for an extra extension such as .txt. If the download link shows 404 Not Found, tell the organizers: the link needs checking. If you open the saved YAML file in a text editor, its first line should be name: carpentries, rather than a web page or error message.
Environment creation is slow or fails
Solving the environment and downloading packages can take several minutes. Wait for the prompt to return. If you receive an error, save the message and contact the organizers. If the message says an environment named carpentries already exists, ask us to check it before removing or replacing it.
conda activate carpentries fails, or jupyter lab is not found
Check which environments are available:
conda env list
If carpentries is listed, try conda activate carpentries again. After activation, an asterisk in the conda env list output should mark it as the active environment. If activation fails, the environment is missing, or JupyterLab still will not start, share the error with the organizers. Do not delete an existing environment to troubleshoot on your own.
JupyterLab does not open in a browser
Keep the terminal open. If it shows a web address beginning with http://localhost: or http://127.0.0.1:, copy the full address into your browser, including any token= portion. Do not share the token. If the browser still cannot connect, or the terminal shows an error instead of an address, contact the organizers.
JupyterLab opens, but the data folder is missing
You may have started JupyterLab from another folder. Stop it using the instructions above. Then follow Open it again on workshop day in your operating system's tab, including the cd command. If the folder is still missing, check that you extracted data.zip into lc-python.