Good source of coding puzzles for programming interviews

Here is a paper which gives a set of coding puzzles which could be useful for technical interviews in data science.

The paper introduces a new type of programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis, and release an open-source dataset of Python Programming Puzzles (P3).

Each puzzle is defined by a short Python program f, and the goal is to find an input x which makes f output True.

Paper: https://bit.ly/3cQcSFj
Problems: https://bit.ly/2THhBCd
Dataset: https://bit.ly/3zAjLEg

Thanks to Dennis Bakhuis (where I say the paper as a LinkedIn post)

List of puzzles is as below

algebra

• Quadratic Root
• All Quadratic Roots
• Cubic Root
• All Cubic Roots

basic     

• Sum Of Digits
• Float With Decimal Value
• Arithmetic Sequence
• Geometric Sequence
• Line Intersection
• If Problem
• If Problem With And
• If Problem With Or
• If Cases
• List Pos Sum
• List Distinct Sum
• Concat Strings
• Sublist Sum
• Cumulative Sum
• Basic Str Counts
• Zip Str
• Reverse Cat
• Engineer Numbers
• Penultimate String
• Penultimate Rev String
• Centered String

chess    

• Eight Queens Or Fewer
• More Queens
• Knights Tour
• Uncrossed Knights Path
• UNSOLVED_Uncrossed Knights Path

classic_puzzles  

• Towers Of Hanoi
• Towers Of Hanoi Arbitrary
• Longest Monotonic Substring
• Longest Monotonic Substring Tricky
• Quine
• Rev Quine
• Boolean Pythagorean Triples
• Clock Angle
• Kirkman
• Monkey And Coconuts
• No Colinear
• Postage Stamp
• Squaring The Square
• Necklace Split
• Pandigital Square
• All Pandigital Squares
• Card Game
• Easy
• Harder
• Water Pouring
• Verbal Arithmetic
• Sliding Puzzle

codeforces        

• Is Even
• Abbreviate
• Square Tiles
• Easy Twos
• Decreasing Count Comparison
• Vowel Drop
• Domino Tile
• Inc Dec
• Compare In Any Case
• Sliding One
• Sort Plus Plus
• Capitalize Firs tLetter
• Longest Subset String
• Find Homogeneous Substring
• Triple
• Total Difference
• Triple Double
• Repeat Dec
• Shortest Dec Delta
• Max Delta
• Common Case
• Five Powers
• Combination Lock
• Combination Lock Obfuscated
• Invert Permutation
• Same Different
• Ones And Twos
• Min Consecutive Sum
• Max Consecutive Sum
• Max Consecutive Product
• Distinct Odd Sum
• Min Rotations

compression     

• LZW
• LZW_decompress
• Packing Ham

conways_game_of_life 

• Oscillators
• Spaceship

games  

• Nim
• Mastermind
• Tic Tac Toe X
• Tic Tac Toe O
• Rock Paper Scissors

game_theory    

• Nash
• ZeroSum

graphs 

• Conway
• Any Edge
• Any Triangle
• Planted Clique
• Shortest Path
• Unweighted Shortest Path
• Any Path
• Even Path
• Odd Path
• Zarankiewicz
• Graph Isomorphism

ICPC     

• Bi Permutations
• Optimal Bridges
• Checkers Position

IMO      

• Exponential Coin Moves
• No Relative Primes
• Find Repeats
• Pick Near Neighbors
• Find Productive List
• Half Tag

lattices 

• Learn Parity
• Learn Parity With Noise

number_theory

• Fermats Last Theorem
• GCD
• GCD_multi
• LCM
• LCM_multi
• Small Exponent Big Solution
• Three Cubes
• Four Squares
• Factoring
• Discrete Log
• GCD
• Znam
• Collatz Cycle Unsolved
• Collatz Generalized Unsolved
• Collatz Delay
• Lehmer

probability         

• Birthday Paradox
• Birthday Paradox Monte Carlo
• Ballot Problem
• Binomial Probabilities
• Exponential Probability

  Image source walmart jigsaw puzzle

Source Prolead brokers usa

The Lost Art of Decile Analysis

                                                                  Image Source: Author

“Logistic Regression is not Regression but a Classification Algorithm”.

You might have seen this in latest popular machine learning books, blogs or you might have heard *Data Science Gurus* utter the same in their highly subscribed YouTube channels.

Machine learning has usurped and renamed many statistical techniques. Often to the extent that they now disbelieve and reject its statistical origins. Case in point is “Logistic regression is not Regression”

However, nothing can be further from the truth than this assertion. We live in a meme culture, memes even become cryptocurrency.

So why not use a meme to drive home the point that Logistic Regression is indeed Regression.

                                                                Image Source: Author

This meme would have made things clearer for most of you. However, let me unpack it for more clarity.

Logistic Regression models a continuous outcome (the probabilities). The probabilities range between 0 and 1.

Binary Logistic Regression is used as a Classification algorithm when we want the response variable to be dichotomous (Churn/Not Churned, Pass/Fail, Spam/No spam etc.)

Usually, we make Logistic Regression into a classification algorithm by setting an appropriate probability cut-off or threshold (0.4, 0.5, 0.6 etc.).

The problem of classifying using a threshold value

Fixing the probability threshold is purely a business call and not a statistical one. The threshold could vary from one domain/industry to another.

Frank Harrell in his blog¹ aptly makes the point “classification is a forced choice”.

Now consider this example, you choose a threshold value of 0.5. Now the ML algorithm outputs the probability of default or no default (1- default, 0 — no default) for 4 customers as 0.51, 0.49, 0.23 and 0.92. Based on the threshold, 2 are classified as ‘default’ and 2 are classified as ‘no default’. However, ask yourself this, isn’t it too close to call for the customers with probability 0.51 and 0.49? 0.51 is surely closer to 0.49 (which is classified as no default) than it is to 0.92 (which is classified as default).

Some popular machine learning packages and low code tools do not depict the predicted probabilities to the user explicitly. The user is hence oblivious to what predicted probabilities are gotten. He /she simply gets the decision — default or no default (1 or 0). In the case of 0.49 and 0.51, the user happily made the decision that the person will not default and will default respectively. But a peek inside the predicted probabilities reveals that it was too close to call!

The other problem with thresholds is that, when we use an improper scoring rule like classification accuracy, it can be easily gamed. For e.g., if out of 100 people, 95 of them default on loan, and 5 don’t. If the classifier classifies everyone to default on loan, then it would have an accuracy of 95% !!

So, is there a better way to use Logistic Regression?

The answer is Yes.

Industries such as Finance and Marketing use Logistic Regression in a more appropriate way for Credit risk modelling and marketing campaign targeting respectively.

A real use case

Let us say, you are a CMO in charge for sales and marketing of a product in your organization. You plan to launch a marketing campaign to increase sales of that product. You have been given a fixed budget for this. Now you would like to get the best ROI possible i.e., spend the exact fixed budget or even lesser and get the maximum sales possible. Here is what you have.

You have the data for 10,000 customers who had either purchased or not purchased a similar product in the past.

You would like to understand which customers should be targeted to increase the likelihood of purchase this time.

You would obviously like to target those people who are more likely to buy the product since you have a fixed budget for the campaign. How to go about it?

What is a Decile Analysis?

Decile analysis was once a popularly used technique, however the convention of teaching and bucketing machine learning problems into either ‘classification’ or ‘Regression’ types, lead people to forget Decile analysis type analyses. I am pretty sure, most freshly minted data scientists would not have even heard of Decile analysis. So, coming back to what is Decile Analysis.

Decile Analysis is used to categorize dataset from highest to lowest values or vice versa. (Based on predicted probabilities)

As obvious from the name, the analysis involves dividing the dataset into ten equal groups. Each group should have the same no. of observations/customers.

It ranks customers in the order from most likely to respond to least likely to respond.

How is Decile Analysis done?

Step 1: Build a Logistic Regression model. In this case the dependent variable was ‘probability of buying the product’. With 1 denoting purchased, 0 denoting not purchased. Also, the relevant independent variables were selected.

Step 2: Get the predicted probabilities from Logistic Regression algorithm. Arrange the probabilities in descending order.

Step 3: Divide the whole dataset into 10 groups, each group should contain equal no. of observations. So, if there 10,000 records, each group would have 1000 records/customers.

Step 4: Compute % of responders for each decile

Step 5: Compute response rate for each decile

Step 6: Compute lift for each decile

The top decile would have customers who are most likely to respond followed by the decile 2 which would have customers who are next most likely to respond and so on.

One major advantage of the decile analysis is that the probabilities and probability range are their own error measures. That is, if the top decile has the probability range 0.75–0.81, then then the probability of this person not buying the product i.e., the error would be (1- [0.75–0.81], here ‘[]’ denotes the interval of values between 0.75–0.81 including 0.75 and 0.81)

What does a Decile Analysis output look like?

The below table depicts a typical decile analysis output.

                                                     Table 1 – Source: Author

Let’s unpack the output table.

As stated earlier, each decile has equal number of customers (1000 in each decile).

% of responders for each decile = No. of responders in that decile/ No. of total responders for all 10 deciles

Refer to Table 1:

Here, % of responders for Decile 1 = 224/984 = 22.8%

984 is the total no. of responders in all 10 deciles

Similarly, % of responders for Decile 2 = 16.5% (162/984)

And cumulative % of responders for top 2 deciles = 39.2%

Gains and Gain Chart

From table 1, Decile 1 contains top 10% of the customers who are most likely to buy. Decile 1 has the highest no. of responders (224 responders).

So out of the total responders in all deciles, 22.8% of the responders are in Decile 1. So, for 10% of the customer base, there are 22.8% responses in Decile 1.

Similarly for 20% of the customer base in decile 1 and decile 2, there are 39.2% responses.

Gain Chart below explains this better.

                                                                              Image Source: Author

Takeaway from Gain Chart: Gain Chart can be used to assess what % of customers respond in each decile. So, rather than targeting customers from lower deciles, customers can be picked up from the top deciles only.

The dashed line is the Baseline. Baseline tells how much % of customers would respond if we target customers randomly without a model.

Response Rate

Response Rate tells what is the percentage of customers who responded in each decile. The response % is highest in Decile 1 followed by Decile 2 and so on.

Response Rate for each decile = No. of respondent in that decile/ No. of customers in that decile

Refer to Table 1:

Here, Response Rate for Decile 1 = 224/ 1000 = 22.4%

Takeaway from Response Rates comparison: Response Rate of each decile is depicted in the chart below. The average response rate of all deciles is 9.8%. So, customers from Decile 1 to Decile 4 lie above the average response rate and should be targeted for the campaign.

                                                            Image Source: Author

Lift and Lift Curve

Lift = cumulative % of responders / customers % at each decile

Refer to Table 1:

Lift for Decile 1 = 22.8%/10% = 2.28

Lift for Decile 2 = 39.2%/20% = 1.96

How to interpret: If we target top two deciles, then we would target 20% of the customers. In the same deciles, the cumulative % of responders are 39.2%. Hence, there is a lift of 1.96.

A lift of 1 means there is no gain compared with the no. of customers targeted at random. Lift greater than 1 means, model approach is better than selecting the customers at random.

Takeaway from Lift Chart: Can be used to identify the deciles which have a higher Lift.

                                                                 Image Source: Author

How to use Decile Analysis in business decision making:

Now that we have done building Decile Analysis, the next pertinent question is how we use it to make effective business decisions:

Let us revisit our Decile Analysis Table

                                                                             Image Source: Author

References:

1. https://www.fharrell.com/post/classification/

Source Prolead brokers usa

More Fun Math Problems for Machine Learning Practitioners

This is part of a series featuring the following aspects of machine learning:

This issue focuses on cool math problems that come with data sets, source code, and algorithms. See previous article here. Many have a statistical, probabilistic or experimental flavor, and some are dealing with dynamical systems. They can be used to extend your math knowledge, practice your machine learning skills on original problems, or for curiosity. My articles, posted on Data Science Central, are always written in simple English and accessible to professionals with typically one year of calculus or statistical training, at the undergraduate level. They are geared towards people who use data but are interesting in gaining more practical analytical experience. The style is compact, geared towards people who do not have a lot of free time. 

Despite these restrictions, state-of-the-art, of-the-beaten-path results as well as machine learning trade secrets and research material are frequently shared. References to more advanced literature (from myself and other authors) is provided for those who want to dig deeper in the interested topics discussed. 

1. Fun Math Problems for Machine Learning Practitioners

These articles focus on techniques that have wide applications or that are otherwise fundamental or seminal in nature.

1. New Mathematical Conjecture?
2. Cool Problems in Probabilistic Number Theory and Set Theory
3. Fractional Exponentials – Dataset to Benchmark Statistical Tests
4. Two Beautiful Mathematical Results – Part 2
5. Two Beautiful Mathematical Results
6. Four Interesting Math Problems
7. Number Theory: Nice Generalization of the Waring Conjecture
8. Fascinating Chaotic Sequences with Cool Applications
9. Representation of Numbers with Incredibly Fast Converging Fractions
10. Yet Another Interesting Math Problem – The Collatz Conjecture
11. Simple Proof of the Prime Number Theorem
12. Factoring Massive Numbers: Machine Learning Approach
13. Representation of Numbers as Infinite Products
14. A Beautiful Probability Theorem
15. Fascinating Facts and Conjectures about Primes and Other Special Nu…
16. Three Original Math and Proba Challenges, with Tutorial
17. Challenges of the week

2. Free books

• Statistics: New Foundations, Toolbox, and Machine Learning Recipes

  Available here. In about 300 pages and 28 chapters it covers many new topics, offering a fresh perspective on the subject, including rules of thumb and recipes that are easy to automate or integrate in black-box systems, as well as new model-free, data-driven foundations to statistical science and predictive analytics. The approach focuses on robust techniques; it is bottom-up (from applications to theory), in contrast to the traditional top-down approach.

  The material is accessible to practitioners with a one-year college-level exposure to statistics and probability. The compact and tutorial style, featuring many applications with numerous illustrations, is aimed at practitioners, researchers, and executives in various quantitative fields.

• Applied Stochastic Processes

  Available here. Full title: Applied Stochastic Processes, Chaos Modeling, and Probabilistic Properties of Numeration Systems (104 pages, 16 chapters.) This book is intended for professionals in data science, computer science, operations research, statistics, machine learning, big data, and mathematics. In 100 pages, it covers many new topics, offering a fresh perspective on the subject.

  It is accessible to practitioners with a two-year college-level exposure to statistics and probability. The compact and tutorial style, featuring many applications (Blockchain, quantum algorithms, HPC, random number generation, cryptography, Fintech, web crawling, statistical testing) with numerous illustrations, is aimed at practitioners, researchers and executives in various quantitative fields.

To receive a weekly digest of our new articles, subscribe to our newsletter, here.

About the author:  Vincent Granville is a data science pioneer, mathematician, book author (Wiley), patent owner, former post-doc at Cambridge University, former VC-funded executive, with 20+ years of corporate experience including CNET, NBC, Visa, Wells Fargo, Microsoft, eBay. Vincent is also self-publisher at DataShaping.com, and founded and co-founded a few start-ups, including one with a successful exit (Data Science Central acquired by Tech Target). He recently opened Paris Restaurant, in Anacortes. You can access Vincent’s articles and books, here.

Source Prolead brokers usa

In a Cloud-Native World, It’s Time to Rethink Data Storage

Digital transformation has created new product and service capabilities and untold additional yottabytes of data. It has become increasingly clear that data is a key creator of value. Take, for example, the realm of digital entertainment. For proof, just scan your monthly credit card bill for streaming service subscriptions. Also, take a moment to think of truly impactful digital content — the MRI that aids a doctor in early disease detection for a patient, the genome data that helps unlock a cure and the convenience of planning our daily lives online for work, family, travel and entertainment.

There has been a corresponding change in how data is created, stored and consumed. People generate data both in their business and personal lives, but we now also see machine-generated data being created at a massive pace in manufacturing locations, utilities, vehicles and so on. Data lives in our homes, cars, on cruise ships, in airplanes, hospitals, sports stadiums and many more places.

Consequently, organizations need to create a plan for infrastructure to consume, manage, store and protect data anywhere. This now translates into data everywhere, from the data center to the cloud and to the emerging “edge” — and this edge is a dramatically growing area of technology innovation and consumption.

Data storage’s level playing field

A decade or two ago, the storage administrator was the employee who managed storage within the enterprise data center. These deeply knowledgeable and technical professionals understand that protecting data is key to their business success and making it consumable to the right people (and only the right people) is the primary objective of their jobs. Understanding how data is stored, its formats and how it is accessed and consumed gave rise to a specialized world of users who understand the speeds and feeds of storage and fluently speak the language of technical data storage acronyms.

As change continues at record pace, it’s no longer just the enterprise IT staff who have the responsibility of capturing, protecting and giving access to data storage. It has become the domain of a broad range of application owners and technical architects as well as highlighted the role of development operations or “DevOps” teams. This collection of people now makes critical decisions within enterprises for solutions — which encompass applications, people, processes and infrastructure — and all of these decisions are made in a more independent manner than before.

Cloud-native shakes things up

Whereas we used to hear about enterprise resource planning (ERP) and business process re-engineering (BPR), we now hear about business applications, data lakes, big data analytics, artificial intelligence and machine learning. These workloads are driving major changes in data, how much of it needs to be stored and how it gets consumed.

Workloads of this type welcome modern design methodologies and principles in application development, design and deployment. This new wave, termed cloud-native, includes the use of distributed software services packaged and deployed as containers and orchestrated on Kubernetes. The promises of this new approach include efficiency, scalability and — very importantly — portability. The latter aspect will allow software applications and infrastructure to support the new dynamic described earlier: data is created and lives everywhere.

That’s the technical aspect of the change. The storage aspect sees that cloud-native applications will also change how storage is accessed, provisioned and managed. This is a world of software services and interactions between services through well-defined interfaces or APIs. Storage has historically been an area where standard interfaces have been adopted. In the realm of file systems, specifically, there are well-known SMB and NFS protocols. 

For cloud-native applications, there is a natural fit of API-based access to storage, which object storage supports naturally through its RESTful APIs. The popular Amazon S3 API is now fully embraced by independent software vendors (ISVs) and storage vendors alike for the cloud, data center and the edge. APIs also apply to storage management and monitoring, and API-based automation is another central theme in this cloud-native wave.

Future-proofing storage

Object storage brings all the right ingredients together – offering portability, API-based access, automation and scalability to effectively unbounded levels – to be the optimal storage model for the new cloud-native world. Next-generation object storage solutions can and will go further in providing higher levels of performance for new applications and workloads and will also provide simplicity of operations to ensure that wider ranges of users will be able to fully exploit them.

Data storage and management have become increasingly complex in the age of apps. Demands shift with the technology, mandating a new method of data management and delivery. Lightweight, cloud-native object storage is what’s needed to power this next generation of cloud-native applications throughout their entire lifecycle – no matter where your data resides.

Source Prolead brokers usa

The Characteristics to Look for in a Custom Web Development Company

If you operate a business, having a fantastic website is no longer a choice. Consumers in the United States spent $517.36 billion on the internet in 2018. Any business that isn’t online is missing out on a huge revenue stream.

Websites are the first point of contact for clients, whether you’re a tiny business or a major corporation. So, how can you create a website that is both engaging and effective in converting leads into sales?

The solution is straightforward. You begin by hiring the correct web designer.

There are hundreds of web development companies for every firm. Several people are available to assist you with your web design needs, ranging from freelancers to established web design organizations.

It can be difficult to narrow down your options with so many web design options available. How do you know that the individual you hired is the proper person, from do-it-yourself websites to long-tenured computer experts?

The first thing to remember is that if you have little to no experience with web design, you should not do it on your own. Those DIY websites may look appealing, but you’re already in over your head unless you know the ins and outs of networking.

The majority of those DIY sites use themes, which require plugins for features like taking payments, scheduling appointments, and so on. You’ll be stuck with a faulty website if one of these plugins isn’t compatible or, worse, allows malware to be installed.

It is always best to hire a web development company from the beginning for this reason alone. To best take advantage of them, keep these steps in mind.

Select Someone With Prior Experience

Just because someone has a large social media following does not mean they can design a website. It’s not sensible to demand proof of experience; it’s expected.

Most well-known web development firms have an online portfolio of clients they’ve worked with, and they’re happy to show off their work.

You might be able to get away with hiring someone who just graduated from design school. This method takes advantage of your web developer’s knowledge of the most up-to-date software and processes.

Recent graduates are also willing to work for less in order to expand their portfolio.

Choose someone who has a good track record.

A reference isn’t just a screenshot of a website they created. References are crucial because they provide insight into what it’s like to work with this individual.

Will they take your wishes into account during the design process, or will they rush you into adding features you don’t need just to increase the price?

If you like a website, write down the designer’s name and give them a call. Allow their reputation to speak for itself, as many business relationships begin through word of mouth.

Select Someone Who Is Within Your Budget

“You get what you pay for,” as the saying goes, and this is especially true when it comes to web development. When it comes to hiring someone, price is so crucial that it is frequently the determining factor.

When it comes to hiring a web development company, you must be realistic about your budget. While it would be ideal to have a website with all of the bells and whistles, that may not be possible.

Allow them to demonstrate what sacrifices you can make right now to help them implement the other alternatives later.

Select Someone You Can Rely On

Because this isn’t just anyone you’re employing to do any work, you need to be able to trust them. This is the person in charge of designing and developing what will become your company’s face.

You’re giving them your image and reputation, as well as a platform to display it to the rest of the world. Your client will not hesitate to blame your web developer if something goes wrong. They’ll link the error messages and 404 pages to you and your company.

This web development firm will also have complete access to your website’s backdoor. Someone untrustworthy holding the keys to a company is the last thing any company needs.

Choose someone who is a good communicator.

Have you ever collaborated on a project with someone who struggled to communicate? It’s the absolute worst. Miscommunication costs businesses not only time but also money.

Keeping up with the project allows you to communicate with your clients about what they may expect. People are happier and more at ease with the process if they know what to expect.

Choose someone who is passionate about what they do.

Although web development is a technical field, its foundation is based on creativity. You want to be able to see that any potential web development company is enthusiastic about their work when you speak with them.

If they lack enthusiasm or zeal, they may not be the best candidate for the job.

Choose someone who is adaptable.

How many times have you been at a meeting and had to rearrange the entire project because one decision was changed? It occurs more frequently than you might imagine, particularly in the web design world.

You may have everything planned out at the start of the project, but then something happens, and you have to remap everything. It happens all the time. However, the last person you need is someone who is so set in their ways that they refuse to change.

It’s All About Collaboration

You are hiring a partner for your project when you hire a web development company. Someone who can see your vision and bring it to reality is required. They must be able to adapt and change to your overall aims as needed. In the end, it’s all about giving your customers an excellent web experience.

Source Prolead brokers usa

Applying Regression-based Machine Learning to Web Scraping

Whenever we begin dealing with machine learning, we often turn to the simpler classification models. In fact, people outside of this sphere have mostly seen those models at work. After all, image recognition has become the poster child of machine learning.

However, classification, while powerful, is limited. There are lots of tasks we would like to automate that are impossible to do on classification. A great example of that would be picking out some best candidates (according to historical data) from a set.

We are intending to implement something similar at Oxylabs. Some web scraping operations can be optimized for clients through machine learning. At least that’s our theory.

The theory

In web scraping, there are numerous factors that influence whether a website is likely to block you. Crawling patterns, user agents, request frequency, total requests per day – all of these and more have an impact on the likelihood of receiving a block. For this case, we’ll be looking into user agents.

We might say that the correlation between user agents and block likelihood is an assumption. However, from our experience (and from some of them being blocked outright), we can safely say that some user agents are better than others. Thus, knowing which user agents are best suited for the task, we can receive less blocks.

Yet, there’s an important caveat – it’s unlikely that the list is static. It would likely change over time and over data sources. Therefore, static-rule-based approaches won’t cut it if we want to optimize our UA use to the max.

Regression-based models are based on statistics. They take two (correlated) random variables and attempt to create a minimal cost function. A simplified way to look at minimal cost functions is to view it as a line that has the least average distance from all data points squared. Over time, machine learning models can begin to make predictions about the data points.

Simple linear regression. There are many ways to draw the line but the goal is to find the most efficient one. Source

We have already assumed with reason that the amount-of-requests (which can be expressed in numerous ways and will be defined later) is somehow correlated with the user agent sent when accessing a resource. As mentioned previously, we know that a small number of UAs have terrible performance. Additionally, from experience we know a significantly larger number of user agents have an average performance.

My final assumption might be clear – we should assume that there are some outliers that perform exceptionally well. Thus, we accept that the distribution of UAs to amount-of-requests will follow a bell-curve. Our goal is to find the really good ones. 

Note that amount-of-requests will be correlated with a whole host of other variables, making the real representation a lot more complex. 

                                                    Intuitively our fit should look a little like this. Source

But why are user agents an issue at all? Well, technically there’s a practically infinite set of possible user agents in existence. As an example, there are over 18 millions of UAs available in one database for just Chrome. Additionally, the number keeps growing by the minute as new versions of browsers and OS get released. Clearly, we can’t use or test them all. We need to make a guess which will be the best.

Therefore, our goal with the machine learning model is to create a solution that can predict the effectiveness (defined through amount-of-requests) of UAs. We would then take those predictions and create a pool of maximally effective user agents to optimize scraping.

The math

Often, the first line of defense is sending the user a CAPTCHA to complete  if he has sent out too many requests. From our experience, allowing people to continue scraping even if the CAPTCHA is solved, results in a block quite quickly.

Here we would define CAPTCHA as the first instance when the test in question is delivered and requested to be solved. A block is defined as losing access to the usual content displayed on the website (whether receiving a refused connection or by other means).

Therefore, we can define amount-of-requests as the amount of requests per given time to a specific source a single UA can make before receiving a CAPTCHA. Such a definition is reasonably accurate without forcing us to sacrifice proxies.

However, in order to measure the performance of any specific UA, we need to know the expected value of the event. Luckily, from the Law of Large Numbers we can deduce that after an extensive amount of trials, the average of the results will converge towards the expected value. 

Thus, all we need to do is allow our clients to continue their daily activities and measure the performance of each user agent according to the amount-of-requests definition.

Since we have an unknown expected value that is deterministic (although noise will occur, we know that IP blocks are based on a specified ruleset), we will commit a mathematical atrocity – decide when the average is close enough. Unfortunately, without data it’s impossible to say beforehand how many trials we need.

Our calculations of how many trials are needed until our empirical average (i.e. the average of our current sample) might be close to the expected value will depend on sample variance. Our convergence of random variables to a constant c can be denoted by:

where the variance is:

From here we can deduce that higher sample variances (2) mean more trials-to-convergence. Thus, at this point it’s impossible to make a prediction on how many trials we would need to approach a reasonable average. However, in practice, getting a grasp on the average performance of an UA wouldn’t be too difficult to track.

Deducing the average performance of an UA is a victory in itself. Since a finite set of user agents per data source is in use, we can use the mean as a measuring rod for every combination. Basically, it allows us to remove the ones that are underperforming and attempt to discover those that overperform.

However, without machine learning, discovering overperforming user agents would be guesswork for most data sources unless there was some clear preference (e.g. specific OS versions). Outside of such occurrences, we would have little information to go by.

The model

There are numerous possible models and libraries to choose from, ranging from PyCaret to Scikit-Learn. Since we have guessed that the regression is polynomial, our only real requirement is that the model is able to fit such distributions.

I won’t be getting into the data feeding part of the discussion here. A more pressing and difficult task is at hand – encoding the data. Most, if not all, regression-based machine learning models only accept numeric values as data points. User agents are strings.

Usually, we may be able to turn to hashing to automate the process. However, hashing removes relationships between similar UAs and can even potentially result in two taking the same value. We can’t have that.

There are other approaches. For shorter strings, creating a custom encoding algorithm could be an option. It can be done through a simple mathematical process of creating:

• A custom base(n), where n is the number of all symbols used.
• Assign each symbol to an integer starting from 0.
• Select a string.
• Multiply each assigned integer by nx-1, where x is the length of the string. 
• Sum the result.

Each result would be a unique integer. When needed, the result can be reversed through the use of logarithms. However, user agents are fairly long strings which can result in some unexpected interactions in some environments.

A cognitively more manageable approach would be to tilt user agents vertically and use version numbers as ID generators. As an example, we can create a simple table by taking some existing UA:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/605.1.15 (KHTML, like Gecko)

You might notice that there’s no “Windows NT” in the example. That’s the important bit as we want to create strings of as equal length as possible. Otherwise, we increase the likelihood of two user agents pointing to the same ID. 

As long as a sufficient number of products is set into the table, unique integers can be easily generated by stripping version numbers and creating a conjunction (e.g. 50101456051150). For products that have no assigned versions (e.g. Macintosh), a unique ID can be assigned, starting from 0.

As long as the structure remains stable over time, integer generation and reversion will be easy. They will likely not result in overflows or other nastiness.

Of course, there needs to be some careful consideration before the conversion is implemented because changing the structure would result in a massive headache. Leaving a few “blind-spots” in case it needs to be updated might be wise.

Once we have the data on performance and a unique integer generation method, the rest is relatively easy. As we have assumed that it might follow a bell-curve distribution, we will likely have to attempt to fit our data into a polynomial function. Then, we can begin filling the models with data.

Conclusion

You don’t even need to build the model to benefit from such an approach. Simply knowing the average performance of the sample and the amount-of-requests of specific user agents, would allow you to look for correlations. Of course, that would take a lot of effort until a machine learning model would be able to do it all for you.

Source Prolead brokers usa

The Role of Big Data in Banking

How do modern banks use Big Data? 

Recently, we have been hearing about Big Data more and more often. In today’s digital world, this technology is being actively used in the financial industry as well. Let’s take a closer look at the tasks tackled by Big Data in banking and the ways it ensures cyber security and increases customer loyalty.

Handling data before and now

Some fifty years ago, a typical bank customer – let’s call him Spencer – walked into a branch in his city, where a cashier met him. The cashier knew his client because he had provided services to Spencer for many years. He knew where Spencer worked and what his financial needs were – and accordingly, he understood how to serve him.

Such a model existed for quite a long time. Banks earned and maintained the trust of their customers who had personal contact with bank employees. 

Today, Spencer may work for an international company that has offices in several countries. It’s quite possible that he will stay in London for two years, then in Berlin for a year, then in Dubai for another two years, and his next stop will be Singapore.

If the old scheme had been in place until now, it would have turned out to be absolutely unadapted for today’s reality. No bank employee would have accurate information about Spencer’s financial affairs or know how to meet his current financial needs.

We live in a world where many industries, including the banking sector, solve issues thanks to a new customer service model. Data Science in banking allows one to continuously analyze and store all information from traditional and digital sources, creating an electronic trail of each client. Here the technology like Big Data comes to the rescue.

What is Big Data? 

Big Data refers to an ever-growing volume of structured and unstructured information of various formats, which belongs to the same context. The main properties of this technology are volume, velocity, variety, value, and veracity.

Such data sets from various sources are beyond what our usual information processing systems can manage. However, major world companies are already using Big Data to meet non-standard business challenges.

According to Reuters, in 2019, the Financial Stability Board issued a report stating the need for vigilant monitoring of how companies use the Big Data tool. The major players including Microsoft, Amazon, eBay, Baidu, Apple, Facebook, and Tencent have vast databases that surely give them a competitive edge. In addition to their core operations, some of these corporations already offer their clients such financial services as asset management, payments, and lending activities.

The importance of Big Data for banks

Thus, non-banking companies can enter the area of financial institutions due to the availability of the necessary data. And what about Big Data in FinTech for the banks themselves?

American Banker has compiled a list of the main trends in the banking sector in the coming decade. Experts call the increasing role of user data one of the most important areas. After all, if the bank can provide the client with the very services and advice they need at that moment it is first-class performance.

Some banks launch AI-powered apps where users can get advice on financial literacy, spending, saving, and investment – and all this based on their personalized requests.

For example, in 2019, Huntington Bank launched the Heads Up app. It sends warnings to clients about the possibility of covering the planned costs in the next period, based on the dynamics of their spending. Subscription billing notifications let the users know when the free trial ends and they are charged a subscription fee. Other notifications signal erroneous withdrawals of amounts from customer accounts, for example, when paying at a store or restaurant.

These applications use Predictive Analytics to monitor transactions in real-time and identify consumer habits, providing them with valuable insights.

Why else is the role of Big Data increasing?

Today, customers don’t have the same attitude toward banks as before. Consider Spencer from our example – earlier, he had to contact the physical branch of the bank to solve each of his issues, and now he can receive an answer to almost any question online.

The role of bank branches is changing. Now they can focus on other important tasks. Clients, in turn, use mobile applications, have constant online access to their accounts, and can perform any operation from their smartphones.

It is also important that, in the modern world, people are more willing to share information about themselves. They leave reviews, mark their location, create accounts on social networks. Such tolerance for risk and willingness to share personal data results in the emergence of a huge amount of information from various channels. This means that the role of Big Data is increasing.

How banks use Big Data

Thanks to the above-described technology, banks can draw conclusions about the segmentation of their customers and the structure of their income and expenses, understand their transaction channels, collect feedback based on their reviews, assess possible risks, and prevent fraud.

Here are just a few examples of how banks use Big Data and what benefits it brings them.

• Analysis of clients’ incomes and expenditures

Banks have access to a wealth of data on clients’ incomes and expenditures. This is information about their salaries for a certain period and the income that passed through their accounts. A financial institution can analyze this information and draw a conclusion about whether the salary has increased or decreased, which sources of income have been more stable, what the expenditure was, which channels the client used to carry out certain transactions.

By comparing the data, banks make informed decisions about the possibility of credit extensions, assess the risks, and consider whether the client is interested in benefits or investments. 

• Segmentation of the customer base

After the initial analysis of the income-expenditure structure, the bank divides its customers into several segments according to certain indicators. This information helps to offer clients the right services in the future. And this means that the financial institution’s employees can better sell auxiliary products and attract customers with the help of individual offers. In addition, the bank can estimate the customers’ expected expenditures and incomes in the next month and draw up detailed plans to ensure the net profit and maximize income.

• Risk assessment and fraud prevention 

Knowing the usual patterns of people’s financial behavior helps the bank to know when something goes wrong. For example, if a “cautious investor” tries to withdraw all the money from their account, this could mean that the card has been stolen and used by fraudsters. In this case, the bank will call the client to clarify the situation.

Analyzing other types of transactions also significantly reduces the likelihood of fraud. For example, Data Science in banking can be used to assess risks when trading stocks or when checking the creditworthiness of a loan applicant. Big Data analysis also helps banks cope with processes that require compliance verification, auditing, and reporting. This simplifies operations and reduces overhead costs.

• Feedback management to increase customer loyalty

Today, people leave feedback on the work of a financial institution by phone or on the website and give their opinion on social networks. Specialists analyze these publicly available mentions with the help of Data Science. Thus, the bank can promptly and adequately respond to comments. This, in turn, increases customer loyalty to the brand.

Today, Big Data analysis opens up new prospects for bank development. Financial institutions that apply this technology better understand customer needs and make accurate decisions. Hence they can be more efficient and prompt in responding to market demands.

Source Prolead brokers usa

Angular vs React: Which is Best for your Business?

When it comes to choosing the right JavaScript framework for developing an exceptional web application, developers have many options. These include Angular, React, Vue, etc. However, it is quite difficult for developers to decide as each of these frameworks has its pros and cons. Therefore, it is highly advisable to compare them before choosing one to implement your project and see which one best suits your needs.

According to the experts, this comparison should be made based on various criteria, which include project size and budget, features to be added to the web application, team expertise, interoperability, etc.

Angular vs React: a brief overview

Let’s take a look at what Angular and React are.

What is Angular?

AngularJS was introduced in 2009, ten years ago, and was adopted by tech giant Google. Angular is an open-source, client-side web framework that helps Angular developers solve problems with single pages developed multiple times. It also helps extend HTML dictionaries by supporting libraries. It gets support from a large community. Angular is still going strong, and after the release of Angular 12, developers can expect its popularity to grow.

What is React?

React was developed by social media king Facebook in 2013. It’s a dynamic open-source JavaScript library that helps create amazing user interfaces. In addition, it also helps in creating single-page apps and mobile apps. The introduction of ReactJS was aimed at providing better and faster speed and making app development easier and more scalable.

ReactJS is typically used in conjunction with other libraries such as Redux. However, when working with a Model View Controller (MVC) architecture, it must be based on V. ReJS. 

The main differences between React and Angular depend on a few aspects. Let’s talk about them separately!

User interface Component

The UI component is one of the factors that differentiates Angular from React. The React community creates its UI tools, and there are many free and paid UI components on the React portal.

Angular, on the other hand, has a built-in material design stack and comes with many pre-developed material design components. Therefore, UI configuration becomes very easy and fast.

Creating Components

AngularJS has a very complex and fixed structure as it depends on three layers: controller, view, and model. Angular allows developers to split the code of an application into multiple files. This allows templates or elements to be used repeatedly in multiple parts of the application.

React, on the other hand, does not choose the same architecture. It provides a simple way to create element trees. The library provides functional programming with declarative component definitions.

React’s code is logically structured and easy to read. They do not require developers to write much code.

Toolset

React uses several tools for code editings, such as Visual Studio, Atom, and Sublime Text. It uses the Create React App tool to run the project, while the Next.js framework is used for server-side rendering. To test an application written in React, developers can use several tools for different components.

Like React, Angular also uses various code editors such as Visual tudio, Aptana, and Sublime Text. Angular CLI helps with project build, while Angular Universal helps with server-side rendering.

However, the main difference between React and Angular is that Angular can be fully tested with a single tool. This can be Jasmine, Protractor, or Karma. And this is the main advantage of Angular over React.

Documentation

In the Angular framework, documentation is not faster because it is a continuous development process. In addition, many of the guides and documentation are still in AngularJS, which is currently useless and outdated for developers.

This is not the case for ReactJS development. React is regularly updated, but knowledge from previous versions is still valuable.

Mobile Application Solutions

For mobile app development, Angular offers the Ionic framework, which includes an interesting library of UI components and the Cordova container. Thus, an application created in the native web application container looks like a web application when viewed on a device.

The same cannot be observed in the ReactJS library. It provides a completely native user interface that helps developers create custom elements and link them with native code written in Kotlin, Java, and Objective-C. This makes React the winner here.

Productivity and development speed

Angular offers a better development experience with its CLI that allows you to seamlessly create workspaces and design working applications and create elements and services using single-line commands, built-in troubleshooting for detailed issues, and the removal of Typescript coding functionality.

When it comes to React, productivity and development speed can be affected by the involvement of third-party libraries. ReactJS developers should opt for the right architecture along with the tools.

In addition, the toolset for React applications varies from project to project, which means that more effort and time should be spent on updating the application if new developers are involved in the project.

This means that Angular wins over React in terms of productivity and speed of app development.

State manipulation

The application uses states in several ways. The user interface of the application is explained by an element at a particular point in time. When the data changes, the framework then re-renders the entire UI element.

In this way, the application ensures that the data is updated. For React, Redux is chosen as the solution for state management, while Redux is not used for Angular.

Popularity

Compared to Angular, React has more searches according to Google Trends. According to the Stack Overflow 2020 Developer Survey, React.js is the most popular and desired web framework among developers.

While people are more interested in Angular because of its many off-the-shelf solutions, both technologies are growing. Therefore, both frameworks are well-known in the market for front-end development.

Freedom and flexibility

A different aspect between Angular and React is flexibility. The React framework offers the freedom to choose the architecture, libraries, and tools for application development. This helps you build a custom app with exactly the set of technologies and features you need, given that you have hired an expert ReactJS development team.

Compared to React, Angular offers limited flexibility and freedom.

Testing

Testing and debugging Angular IO is possible for the entire project using a single tool like Karma, Protractor, and Jasmine. However, the same is not possible for ReactJS application development. Some tools are required to develop different test suites. This maximizes the time and effort spent on the testing procedure. So, in this regard, Angular wins over React.

Binding data

Data binding is another aspect that influences the decision of choosing a suitable framework between Angular and React. React uses unidirectional data binding where UI components can be changed only after the model state has changed. Developers cannot change UI components without updating the corresponding model state.

On the other hand, bidirectional binding is considered for Angular. This method ensures that the model state is automatically changed whenever the UI component is changed and vice versa.

While the Angular approach seems to be more efficient and simpler, the React approach provides a clearer and better data representation for larger application projects. Thus, the winner is React.

Community support

React has more community support than Angular on GitLab and GitHub. According to the StackOverflow 2020 Developer Survey, the number of developers working on React.js projects is higher than the number of developers working on Angular. So React has a lot of community support compared to Android.

Application features and user experience

React uses Fiber and Virtual DOM technologies to build applications, which are the foundation of AngularJS.However, newer versions of Angular boast features and functions such as ShadowAPI that bring deeper competition between the two frameworks without affecting the functionality or size of the app.

Document Object Model (DOM)

Angular uses a true DOM, where the entire tree data structure is current, even if one segment is modified or changed.In contrast, ReactJS uses a virtual DOM that allows application developers to track and update changes without affecting other parts of the tree.In this context, React wins because the virtual DOM is faster than the real DOM.

Is Angular or React better?

Both React and Angular are great tools for app developers. Both frameworks have many advantages, but React works better than Angular so far. It’s being used more and more, it’s trendy and it’s growing. ReactJS also has great support from the developer community.

React outperforms Angular because it performs virtual DOM optimization and rendering. It’s also easy to switch between versions of React. You don’t have to install updates one by one as in the case of Angular. Finally, React offers many solutions to developers to speed up their development and reduce bugs.

No matter what you think about the Angular vs. React comparison, you should make your decision based on your usability and functionality needs.

Angular and React: A Summary

So, from the above general discussion, we can conclude that both of these front-end JavaScript frameworks are readily available and used by web developers all over the world.

Choosing the appropriate framework and maximizing its benefits depends entirely on the requirements of the project.

Angular is a complex framework, while React cannot be considered as a framework but as a library. React.js requires less coding, and if you compare it with Angular based on performance, React.js is likely to be better.

If you are looking for a highly efficient JavaScript developer there are many react development companies and angular development companies that help you in choosing the right fit for your business.

Source Prolead brokers usa

Clickless Analytics is the Future of Business User Analytics

Source Prolead brokers usa

Why was Power BI considered the best BI tool?

Power BI was chosen by Gartner as the best BI tool in the world. This has been happening for the twelfth consecutive year, which reinforces the platform’s power.

When it comes to Business Intelligence and Analytics, no other solution is more recognized in the specialized market than Power BI. This recognition, in addition to being perceived in the market, also occurs among specialists.

In this article, you’ll understand who Gartner is and why your assessment matters. You will also see what criteria were used to choose Power BI as the best BI tool on the market. Follow up!

Who is Gartner, who voted Power BI the best BI tool on the market?

Gartner is the world’s leading information technology (IT) research and consulting firm. It was founded by Gideon Gartner in 1979.

Gartner’s corporate headquarters are located in Stamford, Connecticut. The company has additional offices in other parts of North America as well as in Europe, Asia-Pacific, the Middle East, Africa, Latin America and Japan.

Gartner is so important that today it is publicly traded on the New York Stock Exchange (NYSE) under the stock symbol “IT”. Gartner’s corporate divisions include Research, Executive Programs, Consulting and Events. The company’s two main data visualization and analysis tools are the Magic Quadrant and the Hype Cycle.

Gartner’s Magic Quadrant

The Magic Quadrant is a research visualization and methodology tool to monitor and assess companies’ progress and positions in a specific technology-based market.

Basically, Gartner uses a two-dimensional matrix to illustrate the strengths and differences between companies. The research reports generated help investors find companies that meet their needs and help to compare competitors in their market.

Hype Cycle

The Hype Cycle is a graphical representation that presents the life cycle stages of a technological tool. These stages range from conception to maturity of the solution, which makes it popular in the market.

Hype Cycle stages are often used as reference points in marketing and technology reports. Companies can use the hype cycle to guide technology decisions according to their comfort level with risk.

Why did Gartner choose Power BI as the best BI tool?

Gartner’s recognition that Power BI is the best BI tool on the global market is very important. This is due to the rigorous assessment carried out by this leading IT consulting company. The following are the main reasons why, in the last 12 years, no other Power BI competitor has been chosen by Gartner.

Value

The first criterion evaluated by Gartner is the value that the BI tool offers users. This is easily seen, since users can start using the tool, in its desktop version, for free.

Companies can obtain licenses for specific users at a low cost and then extend to all of their intelligence, always acquiring only the necessary capacity. Power BI is also integrable with legacy tools, which makes it even more functional and quick to transition.

Ease of use

Power BI users can create their own ad-hoc reports in minutes through the familiar drag-and-drop functionality. This is a very practical example of the ease of use of this BI tool. It was also a weighted criterion in Gartner’s evaluation.

Continuous updates and improvement

Regular updates and the addition of new features also weighed heavily on Gartner’s choice of Power BI as the best BI tool. That’s what makes this solution one of the most reliable and, at the same time, highly adaptable to the constant challenges of the corporate world.

In terms of information security, updates and enhancements also prepare the business to respond to evolving risks.

Scope of analysis

Gartner also assesses that no other BI tool has more comprehensive analytics capabilities than Power BI. Platform capabilities allow users to model and analyze to gain valuable business insights.

The architecture is highly flexible, providing integration with other Microsoft solutions as well as third-party tools. All this with fast, simple and secure deployment.

Centralized management

Deployment in seconds is also a Power BI advantage pointed out by Gartner. Companies can deploy in minutes and distribute Business Intelligence content with a few clicks.

In this way, they can leverage the agility of self-service analytics with Power BI IT governance.

global scale

In choosing Power BI as the best BI tool in the world, Gartner also considered that organizations have the flexibility to deploy it wherever they reside, with a global presence.

The platform provides credibility and performance guaranteed by Microsoft, one of the most trusted Information Technology companies in the world.

Security, governance and compliance

Finally, an important criterion evaluated by Gartner is information security. By embracing Power BI, companies have a BI platform that helps them meet stringent standards and certifications. In addition, organizations are assured that they can keep their data secure and that they will control how it is accessed and used.

In summary, Gartner has maintained Power BI as the best BI platform in the global market considering a number of factors that make all the difference to businesses and users. That’s what makes Power BI have more than 5 million users in over 200,000 organizations.

Do you already have the best BI tool in the world? Contact me and see how we can help implement Power BI in your company!

Cleverson Alexandrini Falciano

Source Prolead brokers usa