Record linkage is the process of linking records from different data sources (e.g. pandas dataframes) using any fields in common between them. In this blog post, I’ll talk you through linking records in pandas dataframe using pandas-dedupe. This is a python library build on top of dedupe.io. which links records…

How many times have you found yourself in a situation where you had to deal messy data, especially reconciliate mispellings, short forms of popular names, leading/trailing blanks and so on? I’m talking about something like this: Does this look familiar? It is A-N-N-O-Y-I-N-G, R-I-G-H-T? I have a good suggestion for you…

Pandas is great! You can do pretty much eveything with it: from data cleaning to quick data viz. How about working with nested dictionary from a json file? pandas.json_normalize can do most of the work for you (most of the time). However, json_normalize gets slow when you want to flatten…

Customization and implementation of tips and advice for NER training In this post, I explain how to customize the spaCy Named Entity Recognition (NER) training loop from the comfort of your jupyter notebook, including the implementation of spaCy tips and advice on performance optimization. NB: the code snippets use spaCy…

spacy-annotator is a library to create training data for spaCy Named Entity Recognition (NER) model using ipywidgets. The annotator is fast and, most importantly can leverage existing spaCy models to label your data and pre-fill the annotator for you, even Transformers! GitHub: https://github.com/ieriii/spacy-annotator And when you are ready to give it…

We apply the Universal Language Model Fine-Tuning(ULMFiT) by Howard and Ruder (2018) to fake reviews detection and demonstrate that deep transfer learning outperforms previously researched statistical techniques by Ott, Cardie and Hancock (2013) as well as a standard neural architecture. Additionally, we make several theoretical contributions, including showing that our…