How I Created the UofT Timetable Visualizer
And how you can start developing your own projects too
The UofT Timetable Visualizer is a web-app tool for UofT students to visualize student traffic on campus. In this article, I want to go over my thought process, the hurdles I faced, and the lessons I learned throughout the development of this app. Not only will this serve as a good read for my future self (hi future me!), but I hope it will give insight into the experience of developing a project from start to finish and inspire students to just get started. In developing a project, there are many phases, so I will go through each of them, what to do, and use my experience as an example.
These steps broadly encompass the Software Development Life Cycle (SDLC). Yes, I know we hate project management and just want to start coding, but if you are to create a project from start to finish, software or non-software, you will need to follow these steps. It doesn’t have to be so structured though, in fact I didn’t even realize I was broadly following SDLC until I finished the project!
Step 1: Planning
In this phase, you are figuring out your project idea and how you will execute it. The first question that is always asked is, “What project idea should I do?” It’s honestly a tough question, and there are always basic project ideas listed online that can help get you started. For me, I got into the computer science field to impact and help others, so I was always looking to see if a problem could be fixed by a simple software solution. At my university, I planned a lot of events, and we always wanted to make sure the day and time were optimized so that students were not busy in class during that time. From that, I had the idea of creating an app that displays class timings at UofT to visualize the most busy times on campus.
Once I had the idea, I had to figure out how to implement it. I determined that I needed to do web scraping to get the timetable data from the website, backend development to retrieve the data, and frontend development to display the data. Then I determined the software tools and frameworks best suited for those processes: Python (Selenium) for web scraping, Express/Node.js and MongoDB for the backend, and Vue for the frontend. I took some time to decide on these tools because I didn’t want to start coding and have to change something later on (I ended up switching from Vue to React later on anyways). In terms of order, I wanted to implement my web scraping script first, then implement the frontend and backend that would retrieve and display the data, then deploy it on the web.
The planning phase is also important because it helps validate if our idea is feasible. We may have a crazy idea for a new app, but if we are not advanced enough yet or don’t have enough time for such an endeavor, then it is better for us to realize that earlier on and pivot instead of going head first into coding and then later realize that it isn’t feasible.
Step 2: Designing
Once you have an idea of what you want to do and how you will implement it, you start designing a solution. In larger projects, this means figma designs, UML diagrams, and other prototypes. For my project though, I didn’t need to design too much as it was a simple one-page web app. I used an AI-generated diagram to give me a rough idea where all my components should be, so that I had a base to build on once I started coding. For reference, it looked like this:
Step 3: Developing
This is the crux of the project. Once you have the idea, the plan, and the design, you start coding. I set up the MongoDB database hosted on Atlas, and got to developing the web scraping script. I initially had issues using the ChromeDriver and other Python packages, and ended up using a venv (virtual environment) and having Python auto-download the most recent version of ChromeDriver so that when I deployed, it wouldn’t rely on a downloaded version.
The actual scraping took a lot of time, as the timetable website I scraped from had a lot of detailed intricacies, and I had to navigate to the correct part to scrape. The first-year Computer Science courses that I took really helped me in this, as I had to use complicated for and while loops, nested function calls, and clever break statements (for testing and error handling).
Next, I had to develop the front and backend to retrieve and display the data. I separated my app into components, like the header, footer, filters tab, and actual results view. I configured these components in the App.js file so that they took up the amount of space that they required, then got to working on each separate component. The header and footer were easy, then I got to the filters component, which took me a lot of time. After setting up the frontend buttons for the filters, I had to set up the backend that would load all of the filter options from the database. But then on top of that, I wanted to make the filtering dynamic, such that if a campus option changed, then the associated department and faculty options would also change too. For this, I had to store all the filter data in a structure and dynamically update the options whenever a user changed something.
Once the user selects their filters, I submit the payload to the backend to retrieve the enrolment results. There were many edge cases, like when empty filters were chosen, and they all had to be dealt with. Finally, I was able to send the data back to the frontend, and now I had to display it.
I used ApexCharts options to display the data. I had to format the data correctly, create proper intervals (like dividing the enrolments into 10 parts, each associated with a colour) and options for meaningful visualization. I faced more challenges here, such as certain changes in the payload resulting in the colours on the heatmap turning bleak (this bug still persists as of Sept 2024, so if you are able to fix it, it would be much appreciated!) but in the end, I was able to have a bar chart for the week and for each day, and a heatmap that showed the enrolment distribution for courses at UofT.
I also had to make sure this app was usable on both desktop and mobile, and so that took a lot of time to ensure a specific UI depending on the screen size.
Overall, there were many challenges, such as external package issues, bugs, and undesired behaviour. I would list all the bugs I encountered in developing this project, but then this article would become a book. In short: YOU WILL FACE BUGS. And sometimes it sucks. I spent 2 whole days trying to fix a bug. They are roadblocks, but after the fact, they teach you a lot about how the software works, and you are able to actually do better development because of the knowledge required to fix that bug.
Step 4: Testing
This is the step everyone hates. You just finished developing and now you have to write a bunch of tests that will fail, forcing you to re-write your code. I did a lot of filter-testing, and was able to catch a lot of errors in the way I was filtering for the data which would have resulted in wrong data retrieval.
In terms of UI, I would develop a feature and then see how it looked. If it wasn’t on par with my standards, I would keep trying to fix it until it did. After the fact, I wish I had not been so stringent on specific UI features and just developed the crux of the app, so that I could come back later and fix (or not fix) the smaller issues that took up so much time.
Step 5: Deployment
Finally, my app was working! But now I had to deploy it so that I could access it on the web. I decided to use Oracle Cloud Infrastructure (OCI) to create a virtual machine (VM) I could host my app on.
Already I ran into a bug where for some reason, any downloads would not fully go through. After trying to resolve this for a long time, I read StackOverflow article that listed a bunch of files to remove. I removed all of them and that seemed to fix the issue.
Then I wanted to run my web scraping script on cron so it would run every night, but I ran into issues with ChromeDriver again. I had to update my Python to a newer version to get a newer version of Selenium that would work in my script. Then I had to configure the bash script to use my GitHub credentials so that it could push the results.
Then, I had to host the backend and frontend. I used nginx that would direct backend requests to my backend, and frontend requests to my /build directory. I faced issues here too as the connection was not working, and after a lot of debugging, I had to enable port 80 and 443 for HTTP and HTTPS requests (I dont know why that isnt enabled by default for Oracle VM’s). Then I had to generate an SSL certificate for HTTPS connection.
Finally, the website was up and hosted!
Step 6: Maintenance
Not so fast. Issues always come up in maintenance, especially when data depends on external sources. In my case, I set up error handling in my web script that would detect any issues and halt data collection. My script was working fine, but as of right now (Sept 29, 2024), It is giving me daily errors about some issue with the scraping process. So I have to go check that out.
Additionally, the UofT Timetable website often changes, which results in errors in my script, which I have to fix. It is tedious and a bit annoying, but I’m always motivated by the end impact of the project, and that’s what kept me going throughout the whole process.
Conclusion
Thats it! That is the process of creating a full web application. Though the steps may vary, the process is essentially the same; plan, design, develop, test, deploy, and maintain.
I learned a lot from this project. Web scraping using Selenium, React libraries like ApexCharts, deployment using a VM, and overall full stack development. And importantly, I gained a lot of confidence in my ability to develop full applications that have an impact.
If you are still on the fence about starting a project, just start it. As you’ve read, it is a process full of challenges but also rewards. The best way to learn is to do, and once you can built a small project like this, you are on your way to developing larger apps for real-world applications.
Feel free to reach out to me here if you have any questions about the app, or want to discuss your own project idea!
