Why are people leaving Illinois?

-

 Illinois is famous for being one of the very few states in the country with negative population growth.  The goal of this analysis is to identify the key reasons for the declining population by extracting meaningful insights from unstructured text. The raw text used for this project is in the form of roughly 300,000 news articles and titles scraped from the web that reference Illinois, Chicago, or Springfield. This project was completed in Python & Pyspark with several Natural Language Processing packages. 

After cleaning the raw input text data (removing punctuation, numbers, symbols, etc) , I used latent dirichlet allocation (LDA) to uncover common topics in the data. Put simply, the theory behind LDA is that in a given set of documents, there exist several different latent topics. These topics are latent because we need analytical tools to identify them. Each document is comprised of one or more topics, and each topic is identified by a set of key words. The more often these keywords appear together, the clearer the topic. 

There is no exact scientific solution to LDA, rather an ideal output would make intuitive sense to a human user. That being said, I explored different number of topics with different number of keywords and determined that the nine topic solution made the most sense. 

The topics uncovered in this step of analysis are outlined below:

  1. Marijuana Laws
  2. Illinois COVID19 related news
  3. Illinois' County-level COVID19 case counts & related updates
  4. Chicago Police & crime
  5. Chicago Sports info
  6. Real Estate
  7. Company Market Performance
  8. Miscellaneous Chicago News 
  9. Illinois Public Company News
With these broad topics identified, I then filtered this list to a subset of the topics with which to manually label each news article's text. As you may have noticed, overlap certainly exists between these topics. It is now our job to use this information to create classification buckets and their corresponding rules. The limited topics for this next step were as follows:
  1. Marijuana
  2. COVID19
  3. Crime/Police
  4. Sports
  5. Business/taxes
  6. All Other (Classification Abstained)
This was my personal interpretation of the initial topics above. The goal was to identify classifications as distinct as possible. Keywords were established, and labels were assigned to each observation. If an article contained no keywords or showed no similarity to any of these topics, it was assigned to the "All Other" category. With these classifications assigned, I attempted to confirm its efficacy with a Machine Learning Model. That process followed the below steps:
  1. Text data was converted to numeric format using a CountVectorizer technique. 
  2. This numeric data was split into training & test sets using a 3-1 ratio.
  3. A Logistic Regression model was fit on the training set. 
  4. This model was then used to predict classifications for each observation in the test set. 
The resulting confusion matrix is below:
This model performed quite well, with a 97% overall accuracy score. This step was mainly a proof of concept; are the classifications distinct enough for numerical differentiation. With these classifications assigned, it was time for some sentiment analysis. For this project, I went with a manual dictionary-based approach to sentiment calculation. Roughly two thousand "positive" words and three thousand "negative" words were imported and used to determine overall sentiment scores. The process was simple- inclusion of a positive word means plus one point, negative word means minus one point. Dividing the summation of this process by the number of total words in an article yields the final sentiment score. These scores were then binned into Positive, Negative, and Neutral sentiments. Many other sentiment techniques exist, but experimentation on this dataset yielded the best results from the dictionary-based methodology.

Value counts for these buckets showed the largest proportion falling into the positive sentiment bucket. This was refreshing to find, given the prevalence of divisive media coverage.

Furthermore, a peek into the range of sentiment scores for each classification label  

Legend Explanation

0. Business/Tax

1. Marijuana

2. COVID

3. Crime/Police

4. Sports



Now the fun stuff- Wordclouds. Of the articles deemed to have a positive/negative overall sentiment, what are the most popular words? 

These word clouds can be made slightly more sophisticated by first filtering the data by classification label before splitting off to positive/negative words. For example, here are the results for negative sentiment articles relating to the Coronavirus Pandemic and Crime/Police respectively:

On a lighter note; results from positive sentiment articles related to sports and Illinois business respectively. 


Finally, I performed some targeted sentiment analysis. This analysis was specifically looking for Named Entities that can be identified by an advanced NLP model. These Named Entities can be geographical locations like "Chicago", date-types like "January", companies like "McDonalds", etc. In this analysis, we looks specifically for Organization-Type entities and the sentiment of the language immediately surrounding these organizations. Specifically within the sentence in which an organization is detected, what is the sentiment score? I ran this process on our news articles and compiled the results in the below tables:

This output leads me to the final takeaway from this analysis. Broadly speaking, Illinoisans  are quite proud of their museums, historical/cultural centers, and accomplishments. However, there is significant concern in the broader community regarding violent crime & policing in Chicago. Additionally, government and political entities in the state have strong negative connotations in this data sample. The significant amount of state debt looms large in the minds of residents, and it may be a contributing factor to the annual Illinois population decline. 


Comments

  1. AnonymousDecember 2, 2022 at 3:43 AM

    Pragmatic 메리트카지노 Play is a number one} game developer providing player-favourites to probably the most successful international manufacturers in the iGaming industry. Take mentioned barcode to your nearby CVS Pharmacy, Family Dollar, and even 7-Eleven, and the cashier can scan it, ring you up on the register, and settle for money. It’s not unusual to see plenty of different choices including Ultimate Texas Hold’em, Three Card Poker, Casino Hold’em, Mississippi Stud, and more. This guide goals to provide you with the knowledge you need to|you should|you have to} decide on one of the best gambling website for you. It signifies the expected wager proportion that a game returns to the player over lengthy run|the long term}.

    ReplyDelete

Post a Comment

Popular posts from this blog

Using a Neural Network to Predict Pneumonia From X-Ray Images

-
Image
According to the American Lung Association , ' Pneumonia is an infection of the lungs that may be caused by bacteria, viruses, or fungi. The infection causes the lungs' air sacs (alveoli) to become inflamed and fill up with fluid or pus. That can make it hard for the oxygen you breathe in to get into your bloodstream. The symptoms of pneumonia can range from mild to severe, and include cough, fever, chills, and trouble breathing.' Pneumonia Infographic Patients with pneumonia could have the following symptoms: Cough that produces phlegm or sometimes blood Fever Shortness of breath or difficulty breathing Chills or shaking Fatigue Sweating Chest or muscle pain If a person is suspected of having pneumonia, a doctor will begin by asking the patient about their medical history and symptoms. Next will be a physical exam, so the doctor can listen to the patient's lungs. In checking for pneumonia, a doctor will listen for abnormal sounds like crackling, rumbling or wheezing. I
Read more

Using a Neural Network to Classify Lego Figures

-
Image
 The efficacy of image classification has grown leaps and bounds in recent years with the advent of neural networks. Without going into too much detail, an artificial neural network consists of a series of layers through which inputs flow becoming output predictions. Each layer is composed of a predetermined number of neurons, and each neuron feeds information to the next layer of neurons via weighted connections. These weights are real numbers that flow from layer to layer eventually reaching an output layer, producing a numerical prediction.  For the purposes of image classification, the inputs are pixels. These pixels are converted to numeric inputs by gray-scale figures or color codes (RBG, hex, etc). The output is the neural network's prediction of what that image contains. The neural network learns using a dataset of labeled images; recording correct predictions and attempting to grasp what elements of a picture correspond to which labels.  This abstract concept can be intui
Read more

Artwork Locator

-
Image
 Have you ever wondered where in the world famous artists' work currently resides? I was, so I built this interactive dashboard  (follow the link for the interactive version, below is just a snapshot) to help answer that question. Additionally, the world's largest museums typically only display roughly 5% of their collection at any given time, so I included a graphic outlining the proportion of that artist's work on display vs in storage.  Disclaimer: The data included is a global snapshot gathered in 2016, so changes likely have been made to the locations of certain pieces included in this dataset.
Read more