Benefits of improving data quality

As the digital world continues to become more competitive, everyone is trying to understand their customers better and make finance, development, and marketing decisions based on real data for a better ROI.

Bad data is misleading and would be even more detrimental to your business than a lack of data at all. Organizations may also be forced to abide by data quality guidelines because of compliance issues. If your business’s data is not properly maintained or organized, you may struggle to demonstrate compliance. They find themselves in possession of sensitive personal and financial data such as banks may particularly face more stringent data management prerequisites, if not complied.

Good data quality enables:

Effective decision making: Good quality data leads to accurate and realistic decision-making and also boosts your confidence as you make the decisions. It takes away the need to guesstimate and saves you the unnecessary costs of trials and errors.

More focused: As part of the value chain proposition, it’s critical you know who your prospects are – something that you can only manage analyzing and understanding data. Using high-quality data from your current customer base, you can create user personas and anticipate the needs of the new opportunities and target markets.

Efficient marketing: There are many forms of digital marketing out there, and each one of them works differently for different products in various niches. Good data quality will help you identify what’s working and what’s not.

Better customer relationships: You cannot succeed in any industry if you have poor customer relations. Most people only want to do business with brands they can trust. Creating that bond with your customers starts with understanding what they want.

Competitive Advantage: Being in possession of good quality data gives you a clearer picture of your industry and its dynamics. Your marketing messages will be more specific, and your projections in market changes will bear more accuracy. It will also be easier for you to anticipate the needs of your customers, which will help you beat your rivals to sales.

An AI-augmented data platform, such as DQLabs, would help you detect and address poor data quality issues without the need for much human effort. Since it is AI-based, it will discover patterns and, if possible, tune itself to curb data quality issues of the type it has come across before.

Source Prolead brokers usa

7 Key Advantage of Using Blockchain for Banking Software Development

Did you know!

• Worldwide spending on blockchain solutions is expected to cross 15.9 billion dollars by 2023.
• 90% of U.S. and European banks are exploring blockchain solutions to stay ahead of the game.
• To date, financial institutions alone have spent $552+ million on blockchain-based development projects.

And we can go on and on with these insightful stats relating blockchain with banking and financial institutions, as per Fortunately.

Since getting conceptualized by Satoshi Nakamoto in 2008 in bitcoin cryptocurrency form, blockchain has witnessed remarkable new and innovative applications in software development.

The fintech industry always looks for technology tools that enhance security, and blockchain has emerged as a viable solution. The technology is getting used in diverse ways by banks and other financial institutions to ensure the highest level of privacy and protection.

The infographic below illustrates the blockchain investors’ industry focus for the year 2019.

Source: Statista 

                                                               

With the passing years, the number of deals has increased significantly as diverse industry verticals are exploring the technology. Investment from the banking sector is also rising owing to multiple benefits from the technology.

Blockchain offers a lot more than high-security standards to the financial institution. Have a quick look at the infographic below that lists the top benefits of blockchain for banks and financial institutions.

Before you proceed to learn them in detail, let’s recollect some basic concepts.

So, do you know what exactly defines blockchain?

In simple as ABC, blockchain is a kind of distributed ledger technology that records data in a secured manner with zero possibility of data altercation. Being a distributed technology, it has the following extraordinary features:

 

• Each node of the network keeps the ledger account.
• The data stored is immutable, which means it cannot get modified by a user.
• Every transaction bears a time stamp.
• The data record is encrypted.
• It is a programmable technology.

Main types of blockchain used by the banking industry

The blockchain can be classified into many kinds, but for the sake of lucidity, we will stick to four main types of blockchain:

 

Public Blockchain: A non-restrictive, permission-less distributed ledger system is referred to as the public blockchain. A public blockchain is open to all, and anyone can become an authorized user for accessing past and current records. 

Best examples of Public Blockchain include the digital currency of bitcoin, Litecoin, etc.

 

Private Blockchain: A restrictive and permission-based blockchain is called a private blockchain. It is meant for personal use in enterprises, and the level of scalability, accessibility, gets controlled by the administrative department.

Best examples of Private Blockchain includes Fabric, Corda, etc. 

 

Consortium Blockchain: A blockchain similar to private blockchain but gets utilized by multiple enterprises is called a consortium blockchain. It allows users from multiple enterprises thus actively used by banks, government organizations, etc.

Best examples of Consortium Blockchain includes R3, Energy web foundation, etc.

 

Hybrid Blockchain: As the name suggests, a hybrid blockchain is a mix between Private Blockchain and Public Blockchain. It allows users to go for both permission-based and permission-less features. The organization can control whether to let a particular transaction go for public or private use.

The best example of Hybrid Blockchain is Dragonchain.

 

Here are the top 7 benefits of blockchain solutions in banking software development

1. Reduces Running Cost

Blockchain effectively builds trust between the bank and its trading partner (whether a client or another bank). The high trust between the partners conducting financial transactions removes the necessity of mediators and third-party software otherwise required in the absence of blockchain.

The immutable version of the transactions minimizes the corruption level, thus boosting the confidence level among the users.

 

2. Lightning Speed Transactions

The technology is responsible for effectively reducing the transaction time. It has been possible because it cuts and eliminates multiple intermediaries out of the process.

The result is a simplified transaction with little to zero intermediaries. Also, the trades are conducted with ledger entries which facilitate banks to instantly authorize and permit processes with the least time gap.

3. High-Security Standards

Lightning-speed transactions of blockchain significantly reduce the time span for hackers to divert or hack them. It also allows zero modification powers to the concerned parties, enhancing the transparency level for the users.

It has become feasible because blockchain stores data in a decentralized and encrypted manner over the entire network. It means as soon as the data gets stored over the network, a hacker cannot conduct any alteration to it. 

Any data altercation invalidates the signature, which enhances the security level.

4. Smart contracts improving data handling

Banks and other financial institutions hire app developers for developing smart contracts using blockchain. The technology allows the development team to ensure automatic data verification and quick execution of commands and processes.

It improves the data handling capacity of the developed software with high security and minimum human interference.

 

5. Offers High Accountability

Every transaction conducted online gets duplicated over the entire network. It automatically eliminates the risk of losing transaction details or loss of data. The user can conveniently trace any executed transaction.

So, banks find it very easy to trace and deal with any issues occurring with transactions. Finding the culprit becomes a matter of clicks in such a scenario.

 

6. Regarded as the future of banking software

As per AI development stats, 20+ nations worldwide have researched for developing a national cryptocurrency. With multiple countries formally and informally approving bitcoin trade, digital currencies are leading significant impacts in trade and commerce.

Blockchain is regarded among the most disruptive technologies and has become an integral part of banking software development worldwide. 

 

 7. Improves Efficiency 

As per CEO of IBM, Ginni Rometty, “Anything that can conceive of as a supply chain, blockchain can vastly improve its efficiency- it doesn’t matter if its people, numbers, data, money.”

Convenient tracking of fraudulence, quicker transactions, heavy security, etc., all together aids in developing a positive work culture environment for the bank employees. 

Blockchain improves the efficiency and reliability of the developed software and acts as a positive energy booster for the bank employees.

 

Final Words

That was about the top 7 benefits of using blockchain for banking software. For developing advanced banking apps, hire blockchain developers from India at an affordable hourly rate.

Source Prolead brokers usa

Workplace Flexibility is the New Norm

Source Prolead brokers usa

Web development trends for 2021 and the latest web technology stack

            

Standards in web development can change faster than they can implement. To stay one step ahead, it is essential to keep an eye on the prevailing trends, techniques, and approaches.

We’ve analyzed trends across the industry and put together a definitive list of web development trends for 2021. As a bonus, you’ll also read about the best web development stacks to watch out for next year. Whether your current interest is market development, startup innovation, or IoT invention, these are the trends you need to know about.

The hottest web technology trends to adopt in 2021

“Knowing what the next big trends are will help you know what you need to focus on.”

Mark Zuckerberg

When a billionaire with decades of experience in an industry tells us to do this, we can only agree. Here we’ve put together a list of the top trends to watch out for when growing your web business, so they’re easy to find, save you time, and help you grow your business in the new decade.

1| Voice search

We are currently experiencing the beginning of the era of voice search. Every smartphone equips with a digital voice assistant (Siri for iPhones, Google Assistant for Android devices). Intelligent speakers with artificial intelligence are also becoming more and more popular.

What are the reasons for the move to voice interfaces?

| Ease of use

Communication doesn’t have learners, which means that even children and the elderly can get to grips with voice interfaces without training.

| Accessibility

Digital voice assistants are already a common feature of smartphones. Smartphones are still not very popular, but pricing from $50 is a prerequisite for expansion.

The report states that “the use of voice assistants is reaching a critical mass. And by 2021, some 123 million U.S. citizens, or 37% of the total population, are expected to be using voice assistants.

| It’s good for your business.

Voice search is a big trend in ecommerce. But it applies to all businesses on the web too. If you want people to find your web app, optimize it for voice search as soon as possible.

Consider developing your smart speaker app and help you build a loyal audience, and give you another channel to generate sales.

2|  WebAssembly

When building web applications, performance inevitably suffers: heavy computation slows down due to the limitations of JavaScript, which has a significant impact on the user experience. For this reason, most popular games and powerful applications are only available as native desktop applications.

WebAssembly has emerged to change the game. This new format aims for native performance across web applications; WebAssembly can compile code from any programming language into bytecode that runs in the browser.

3| Content personalization through machine learning

Artificial intelligence, including machine learning, influences our daily online activities without us even realizing it, which is the essence of ML and the experience it provides in our native language.

Machine learning is the ability of software to improve its performance without the direct involvement of the developer. Essentially, the software analyses the input data, discover patterns, makes decisions and improves performance.

For example, Airbnb hired a machine to personalize guest search results to increase the likelihood that hosts would accept a request. A machine learning algorithm analyses each host’s decision to make a bid. In our A/B testing, we saw a 3.75% increase in conversions. A/B testing showed a 3.75% increase in conversions, and as a result, customer satisfaction and revenue increased as the algorithm applied to all Airbnb users.

Netflix’s engineers wanted to go even further. They used a more advanced ML-based algorithm to personalize content to meet the needs of users better predictively. Rather than targeting entire segments of users, each user identifies individually. The algorithm provides content and search results based on the user’s intent, not on previous queries.

A great example, but there are many more. You can improve the user experience by incorporating natural language processing and pattern recognition—machine perception, where a computer interprets data and makes decisions.

4|  Data security

The more data a web application processes, the more attractive it is to cybercriminals. They aim to ruin your services and steal your users’ data and your company’s internal information. Such actions can seriously damage a company’s reputation and cause significant losses.

The security of web services should be a top priority. So here are four things you can do to keep your user data safe in 2021.

| Don’t neglect security testing

Security testing should be carried out during the development phase to prevent data breaches. Should test each change to your web app should be tested explicitly.

| Use a website monitoring tool.

Website monitoring tools allow you to monitor all requests and detect and identify suspicious activity constantly. Timely notifications enable your team to react immediately and protect your web app.

| Choose third-party services carefully.

SaaS software is becoming increasingly popular as it makes app development more accessible and quicker. However, it would be best to make sure that the service provider you work with is reliable.

| Encryption of sensitive data

Even if criminals have access to your database, they won’t be able to extract any use-value from the sensitive data stored there.

Along with these tips, here are the latest trends for the web in 2021 to help you keep your apps and data safe. Must address two essential elements must be addressed here.

| A.I. for cybersecurity

Machines are becoming more intelligent. There are both good and bad sides to this fact, but we will focus on the benefits A.I. brings in this report.

In 2021, we expected A.I. technology to become even more helpful in terms of data security. We have already had the opportunity to look after some of the latest improvements. AI-powered biometric logins that scan fingerprints and retinas are more than just an element of science fiction. The web systems of many powerful companies demonstrate these capabilities.

80% of telecom company founders say they rely on A.I. for cybersecurity.

Threats and malicious activity can be easily detected by AI-powered security software. The more types of malware there are, the more powerful and dangerous they become. That’s why large companies are now training A.I. systems to analyze behavioral patterns in their networks and respond immediately to any suspicious activity.

“Artificial intelligence is an advisor. It’s like a trusted advisor that you can ask questions to. Artificial intelligence learns as it goes. The cognitive system starts learning as we teach it, but it’s there and it doesn’t forget what it’s been taught, so it can use that knowledge to make more accurate decisions when it’s looking at what’s happening from a (security) threat perspective,” said IBM Security Vice President, Kevin Skapinetz, explains how A.I. can influence security systems and save companies from potential threats.

| Blockchain for cybersecurity

Over the last few years, Bitcoin and other blockchain-related topics have dominated tech blogs and reports, and in 2021 we recommend taking a closer look at this tool for the security of web solutions.

NASA has implemented blockchain technology to protect its data and prevent cyber attacks on its services. The starting point is that if influential leaders are using this trend to protect their entities, why can they ignore this principle?

| Database security

Storing all your data in one place makes it perfectly convenient for hackers to steal it. Blockchain is a decentralized database, which means no single authority or location for storing data. All users are responsible for validating the data and can make no changes of any kind without everyone’s approval.

| Protected DNS

DDoS attacks plague large companies. However, there is a cure: a fully decentralized DNS. When content distributes across many nodes, it becomes almost impossible for an attacker to find and use sensitive points or attack a domain.

Security trends in web development

| Making it work for your business

No matter what kind of web app you plan to launch, its security is the number one thing you need to focus on. Look at the most effective approaches and make sure your development team is proficient in security questions and has the skills to keep your critical data safe.

5|  Progressive Web Apps (PWAs) and Accelerated Mobile Pages (AMPs)

Google prioritizes web apps that load quickly on mobile devices. For this reason, you should consider implementing PWAs and AMPs, which are proprietary technologies that reduce the loading time of web pages.

Progressive web apps (PWAs) are web pages that replicate the native mobile experience. PWAs are fast, work online and even with poor internet connections, and are relatively inexpensive. PWAs support interaction, so users are not aware that they are using a browser. E-commerce web applications are an everyday use case for this technology.

AMP (Accelerated Mobile Page) only supports static content, but it loads faster than regular HTML. AMP omits all decorative elements and displays only the necessary information, such as text and images. This approach is ideal for blogs and news publishers.

Whether you should use a PWA or AMP depends on your case. However, it would be best if you started considering these technologies now. You have the opportunity to dramatically improve your search result rankings while providing a high-end experience.

We use so many great PWAs that we don’t need to mention them: some, like Twitter, have over 330 million active users every month, while others are on the verge of spectacular success. If you’re planning to build a simple web game, a lifestyle or sports app, an entertainment app, or a news site, you should consider a PWA approach.

AMP is an excellent idea if

• Most of the users of your web app are accessing it on a mobile device.
• The page loads slowly, and users leave the site quickly.
• You are putting a lot of effort into SEO and app promotion.

6| . Multi-experience

The story of app development started with smartphones, tablets, and laptops. Today, they are so prevalent that it’s hard to imagine a day without a smartphone. More recently, other intelligent devices, cars, smartwatches, and components of IoT systems have gained remarkable popularity. Mobile-friendly apps are a must. However, a fresh and vivid trend is emerging in app development. Multiple experiences are welcome. The idea is to allow users to use your app wherever they want – on their tablet, smartwatch, car, etc. The point is to create apps that look good, work well, and bring value in an equally engaging and helpful way on all devices.

Multi-experience is the new trend in web development.

The trend for 2021 is to make web applications compatible with all screens. According to Gartner, it’s one of the top trends in technology. The traditional idea of using laptops and smartphones to interact with software applications is drifting towards a multi-sensory, multi-touch, multi-screen, multi-device experience. Customers expect apps with great intelligent chatbots, voice assistants, AR/VR modules, etc., to be available on all devices. For web businesses that want to be successful in 2021, this multi-channel approach to human-machine interaction in web apps will be critical.

“This ability to communicate with users across the many human senses provides a richer environment for delivering nuanced information,” said Brian Burke, Research Vice President at Gartner.

5G technology and edge computing will stimulate the development of multi-experience trends. A new generation of wireless technologies will increase transmission speeds and open up opportunities to deliver superior AR/VR experiences. I.T. companies worldwide are poised to provide seamless experiences on mobile and web extensions, wearables, and conversational devices.

7| Motion U.I.

Motion design is one of the key trends in web design for the coming year. Minimalist design combined with sophisticated interaction looks tremendous and grabs the user’s attention.

Think page header transitions, nice hovers, animated charts, animated backgrounds, and modular scrolling. These and many other elements will display your unique style, entertain users, improve behavioral factors and help your web app rank higher in search results.

Use for business

To increase user engagement and provide a better UI/UX for your web app, upgrade it with Motion U.I. technology.

Guide users through your app with animations that show the next steps.

React to the user’s gestures with catchy animations.

Show the relationship between the different components of your app, for example.

Conclusion

Fashions change so quickly that it can be challenging to keep up with them. But why not give it a go?

By keeping up with the latest trends in web development, you can delight your users with a world-class experience, improve your web app’s ranking and open up new markets for your services.

Over the next few years, voice search will strengthen its position, and service providers will adapt to the new reality. By approaching it smartly, you can be the first company to win customers with voice search. Sounds good, right?

The security of user data has been an issue for quite some time now. If you want to become a market leader, you can’t ignore this issue.

By offering a multi-faceted experience to your web app users, you increase your chances of being chosen first by them. So be in touch with top web development companies in India to have a world-class experience.

Source Prolead brokers usa

DSC Weekly Digest 26 April 2021

Source Prolead brokers usa

What is Liskov’s Substitution Principle?

In this article, we will explore the Liskov’s substitution principle, one of the SOLID principles and how to implement it in a Pythonic way. The SOLID principles entail a series of good practices to achieve better-quality software. In case some of you aren’t aware of what SOLID stands for, here it is:

• S: Single responsibility principle
• O: Open/closed principle
• L: Liskov’s substitution principle
• I: Interface segregation principle
• D: Dependency inversion principle

The goal of this article is to implement proper class hierarchies in object-oriented design, by complying with Liskov’s substitution principle.

Liskov’s substitution principle

Liskov’s substitution principle (LSP) states that there is a series of properties that an object type must hold to preserve the reliability of its design.

The main idea behind LSP is that, for any class, a client should be able to use any of its subtypes indistinguishably, without even noticing, and therefore without compromising the expected behavior at runtime. That means that clients are completely isolated and unaware of changes in the class hierarchy.

More formally, this is the original definition (LISKOV 01) of LSP: if S is a subtype of T, then objects of type T may be replaced by objects of type S, without breaking the program.

This can be understood with the help of a generic diagram such as the following one. Imagine that there is some client class that requires (includes) objects of another type. Generally speaking, we will want this client to interact with objects of some type, namely, it will work through an interface.

Now, this type might as well be just a generic interface definition, an abstract class or an interface, not a class with the behavior itself. There may be several subclasses extending this type (described in Figure 1 with the name Subtype, up to N). The idea behind this principle is that if the hierarchy is correctly implemented, the client class has to be able to work with instances of any of the subclasses without even noticing. These objects should be interchangeable, as Figure 1 shows:

Figure 1: A generic subtypes hierarchy

This is related to other design principles we have already visited, like designing for interfaces. A good class must define a clear and concise interface, and as long as subclasses honor that interface, the program will remain correct.

As a consequence of this, the principle also relates to the ideas behind designing by contract. There is a contract between a given type and a client. By following the rules of LSP, the design will make sure that subclasses respect the contracts as they are defined by parent classes.

Detecting LSP issues with tools

There are some scenarios so notoriously wrong with respect to the LSP that they can be easily identified by the tools such as mypy and pylint.

Using mypy to detect incorrect method signatures

By using type annotations, throughout our code, and configuring mypy, we can quickly detect some basic errors early, and check basic compliance with LSP for free.

If one of the subclasses of the Event class were to override a method in an incompatible fashion, mypy would notice this by inspecting the annotations:

class Event:

    …

    def meets_condition(self, event_data: dict) -> bool:

        return False

 

class LoginEvent(Event):

    def meets_condition(self, event_data: list) -> bool:

        return bool(event_data)

When we run mypy on this file, we will get an error message saying the following:

error: Argument 1 of “meets_condition” incompatible with supertype “Event”

The violation to LSP is clear—since the derived class is using a type for the event_data parameter that is different from the one defined on the base class, we cannot expect them to work equally. Remember that, according to this principle, any caller of this hierarchy has to be able to work with Event or LoginEvent transparently, without noticing any difference. Interchanging objects of these two types should not make the application fail. Failure to do so would break the polymorphism on the hierarchy.

The same error would have occurred if the return type was changed for something other than a Boolean value. The rationale is that clients of this code are expecting a Boolean value to work with. If one of the derived classes changes this return type, it would be breaking the contract, and again, we cannot expect the program to continue working normally.

A quick note about types that are not the same but share a common interface: even though this is just a simple example to demonstrate the error, it is still true that both dictionaries and lists have something in common; they are both iterables. This means that in some cases, it might be valid to have a method that expects a dictionary and another one expecting to receive a list, as long as both treat the parameters through the iterable interface. In this case, the problem would not lie in the logic itself (LSP might still apply), but in the definition of the types of the signature, which should read neither list nor dict, but a union of both. Regardless of the case, something has to be modified, whether it is the code of the method, the entire design, or just the type annotations, but in no case should we silence the warning and ignore the error given by mypy.

Note: Do not ignore errors such as this by using # type: ignore or something similar. Refactor or change the code to solve the real problem. The tools are reporting an actual design flaw for a valid reason.

This principle also makes sense from an object-oriented design perspective. Remember that subclassing should create more specific types, but each subclass must be what the parent class declares. With the example from the previous section, the system monitor wants to be able to work with any of the event types interchangeably. But each of these event types is an event (a LoginEvent must be an Event, and so must the rest of the subclasses). If any of these objects break the hierarchy by not implementing a message from the base Event class, implementing another public method not declared in this one, or changing the signature of the methods, then the identify_event method might no longer work.

Detecting incompatible signatures with pylint

Another strong violation of LSP is when, instead of varying the types of the parameters on the hierarchy, the signatures of the methods differ completely. This might seem like quite a blunder, but detecting it might not always be so easy to remember; Python is interpreted, so there is no compiler to detect these types of errors early on, and therefore they will not be caught until runtime. Luckily, we have static code analyzers such as mypy and pylint to catch errors such as this one early on.

While mypy will also catch these types of errors, it is a good idea to also run pylint to gain more insight.

In the presence of a class that breaks the compatibility defined by the hierarchy (for example, by changing the signature of the method, adding an extra parameter, and so on) such as the following:

# lsp_1.py

class LogoutEvent(Event):

    def meets_condition(self, event_data: dict, override: bool) -> bool:

        if override:

            return True

        …

pylint will detect it, printing an informative error:

Parameters differ from overridden ‘meets_condition’ method (arguments-differ)

Once again, like in the previous case, do not suppress these errors. Pay attention to the warnings and errors the tools give and adapt the code accordingly.

Remarks on the LSP

The LSP is fundamental to good object-oriented software design because it emphasizes one of its core traits—polymorphism. It is about creating correct hierarchies so that classes derived from a base one are polymorphic along the parent one, with respect to the methods on their interface.

It is also interesting to notice how this principle relates to the previous one—if we attempt to extend a class with a new one that is incompatible, it will fail, the contract with the client will be broken, and as a result such an extension will not be possible (or, to make it possible, we would have to break the other end of the principle and modify code in the client that should be closed for modification, which is completely undesirable and unacceptable).

Carefully thinking about new classes in the way that LSP suggests helps us to extend the hierarchy correctly. We could then say that LSP contributes to the OCP.

The SOLID principles are key guidelines for good object-oriented software design. Learn more about SOLID principles and clean coding with the book Clean Code in Python, Second Edition by Mariano Anaya.

Source Prolead brokers usa

Question Answering based on Knowledge Graphs

Why Question-Answering Engines?

The search only for documents is outdated. Users who have already adopted a question-answering (QA) approach with their personal devices, e.g., those powered by Alexa, Google Assistant, Siri, etc., are also appreciating the advantages of using a “search engine” with the same approach in a business context. Doing so allows them to not only search for documents, but also obtain precise answers to specific questions. QA systems respond to questions that someone can ask in natural language. This technology is already widely adopted and now rapidly gaining importance in the business environment, where the most obvious added value of a conversational AI platform is improving the customer experience.

Another key tangible benefit is the increased operational efficiency gained by reducing call center costs and increasing sales transactions. More recently we have seen a strong developing interest in in-house use cases, e.g., for IT service desk and HR functions. What if you didn’t have to painstakingly sift through your spreadsheets and documents to extract the relevant facts, but instead could just enter your questions into your trusty search field?

This is optimal from the user’s point of view, but transforming business data into knowledge is not trivial. It is a matter of linking and making all the relevant data available in such a way that all employees—not just experts—can quickly find the answers they urgently need within whichever business processes they find themselves.

With the power of knowledge graphs at one’s disposal, enterprise data can be efficiently prepared in such a way that it can be mapped to natural language questions. That might sound like magic, but it’s not. It is actually a well-established method to successfully roll out AI applications like QA systems in numerous industries.

Where do Current Question-Answering Methods Fall Short?

The use of semantic knowledge graphs supports a game-changing methodology to construct working QA engines, especially when domain-specific systems are to be built. Current QA technologies are based on intent detection, i.e., the incoming question must be mapped to some predefined intents. A common example of this is an FAQ scenario, where the incoming question is mapped to one of the frequently asked questions. This works well in some cases, but is not well suited to access large, structured datasets. That is because when accessing structured data, it is necessary to recognize domain-specific named entities and relations.

In these situations, intent detection technology requires a lot of training data and struggles to provide satisfactory results. We are exploiting a different technology based on semantic parsing, i.e., the question is broken down into its fundamental components, e.g., entities, relations, classes, etc., to infer a complete interpretation of the question. This interpretation is then used to retrieve the answer from the knowledge graph. What are the advantages?

• You do not need special configuration files for your QA engine—everything is encoded within the data itself, i.e., in the knowledge graph. By doing so you automatically increase the quality of your data, with benefits for your organization and for applications using this data.
• Contemporary QA engines frequently struggle with multilingual environments because they are typically optimized for a single language. With knowledge graphs in place, the expansion to additional languages can be established with relatively little effort, since concepts and things are processed in their core instead of simple terms and strings.
• This technology scales, so it will not make a difference if you have 100 entities or millions of entities in your knowledge graph.
• Lastly, you do not need to create a large training data corpus before setting up your engine. The data itself suffices and you can fine-tune the system as you go with little additional training data!

Building QA engines on knowledge graphs: an example from HR

What follows is a step-by-step outline of a methodology using a typical human resources (HR) use case as a running example.

Step 1: Gather your datasets
In this step, business users define the requirements and identify the data sources for the enterprise’s knowledge. After collecting structured, semi-structured and unstructured data in different formats, you will be able to produce a data catalog that will serve as the basis for your enterprise knowledge graph (EKG).

Step 2: Create a semantic model of your data
Here your subject matter experts and business analysts will define the semantic objects and design the semantic schemes of the EKG, which will result in a set of ontologies, taxonomies, and vocabularies that precisely describe your domain.

Step 3: Semantify your data
Create pipelines to automatically extract and semantify your data, i.e., annotate and extract knowledge from your data sources based on the semantic model that describes your domain. This is performed by data engineers who automate the ingestion and normalization of data from structured sources, as well as automate the analysis of unstructured content using NLP tools in order to populate the EKG using the semantic model provided. The resulting enriched EKG continuously improves as new data is added. The result of this step is the initial version of your EKG.

Step 4: Harmonize and interlink your data
After the previous step, your data is represented as things rather than strings. Each object gets a unique URI for links between entities and datasets to be established. This is facilitated by the use of ontologies and vocabularies, which, in addition to mapping rules, allow interlinking to external sources. During this stage, data engineers establish new relations in the EKG using logical inference, graph analysis or link discovery—altogether enriching and further extending the EKG. The result of this process is an extension of your EKG that is eventually stored in a graph database which provides interfaces for accessing and querying the data.
Step 5: Feed the QA system with data
Allowing to ask questions on top of a EKG requires that (a) the data is indexed and (b) ML models are available to understand the questions. Both steps are fully automated in QAnswer. The EKG data is automatically indexed, and pretrained ML models are already provided so that you can start asking questions on top of your data right away.

Step 6: Provide feedback to the QA system
Improving the quality of the answers is done in the following two steps (6 and 7). The business user and a knowledge engineer are responsible for tuning the system together. The business user expresses common user requests and the knowledge engineer checks if the system returns the expected answers. Depending on the outcome, either the EKG is adapted (following Step 2-4) or the system is retrained to learn the corresponding type(s) of questions. 
The user can provide feedback to the provided answer either by stating whether it is correct or not or by selecting the right query from a list of suggested SPARQL queries:

Step 7: Train the QA system
New ML models are generated automatically based on the training data provided in step 6. The system adapts to the type of data that has been put into the EKG and the type of questions that are important for your business.  The provided feedback improves the ML model in order to increase the accuracy of the QA system and the confidence of the provided answers:

Step 8: Gain immediate insight into your knowledge
With the HR dataset now at your fingertips, you can ask questions like the following: Who are my employees? What languages do my staff speak? Who knows Javascript? Who has experience as Project Leader? Who can program in Java and knows MySQL? Who speaks English and Chinese? Who knows both Java and SPARQL? What is the salary range of my employees? How many people can code in Java and Javascript? What is the average salary of a C++ programmer? Who is the top paid employee?

Looking to the future

In order to have a conversation with your Excel files and the rest of the disparate data that has accumulated over the years, you will need to begin by breaking up the data silos in your organization. While the EKG will help you dismantle the data silos, the Semantic Data Fabric solution allows you to prepare the organization’s data for question answering. This approach combines the advantages of Data Warehouses and Data Lakes and complements them with new components and methodologies based on Semantic Graph Technologies.

A lot of doors will open for your company by combining EKGs and QA technologies, and several domain-specific applications that allow organizations to quickly and intuitively access internal information can also be built on top of our solution.

One of the challenges we address is the difficulty of accessing internal information fast, intuitively and with confidence. People can find and gather useful information as they normally would when asking a human—in natural language. The capabilities of the technology we have presented in this article go well beyond what can be achieved with today’s mainstream voice assistants. This new direction offers organizations a significant opportunity to simplify human-machine interaction and profit from the improved access to the organizations’ knowledge while also offering new, innovative and useful services to their customers.

The future of question-answering systems is in leveraging knowledge graphs to make them smarter.

Try our live demo!

Source Prolead brokers usa

Mitigating Emerging Cyber Security Threats Using Artificial Intelligence

Last week, I taught a cybersecurity course at the University of Oxford case. I created a case study for my class based on an excellent recent paper: Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defences (link below)    

 

This paper is unique because it discussed emerging cyber security threats and their mitigation using artificial intelligence in context of advanced autonomous

driving systems (ADSs). I felt that this is significant because typically the problem domain of AI and cybersecurity is mostly an Anomaly detection or a Signature detection problem. Also, most of the times, cybersecurity professionals use specific tools such as splunk or darktrace(which we cover in our course) – but these threats and their mitigations are very new. Hence, they need exploring from first principles/research. Thus, we can cover newer threats such as adversarial attacks(making modifications to input data to force machine-learning algorithms to behave in ways they’re not supposed to). By considering a complex and emerging problem domain like ADASS we can discuss many more emerging problems which we have yet to encounter at scale.

 

A deep learning-based ADS is normally composed of three functional layers, including a sensing layer, a perception layer and a decision layer, as well as an additional cloud

service layer.

 

The sensing layer: comprises heterogeneous sensors such as GPS, camera, LiDAR, radar and ultrasonic sensors are used to collect real-time ambient information including the current position and spatial-temporal data (e.g. time series image frames).

 

The perception layer contains deep learning models to analyze the data collected by the sensing layer and then extract useful environmental information from the raw data for further process.

 

The decision layer acts as a decision-making unit to output instructions concerning the change of speed and steering angle based on the extracted information from

the perception layer.

 

The perception layer includes functions like Localization, Road object detection and semantic segmentation which uses a variety of deep learning algorithms. The cloud service provides compute intensive resources such as preroute planning and enhance the perception of the surrounding environment. The decision layer includes functions like Path planning and object trajectory prediction; Vehicle control via deep reinforcement learning;  

End-to-End driving:

 

These are depicted below

 

 

Based on this, the paper explores the below

ATTACKS IN ADSS

• Physical attacks on sensors
• Jamming attack, Spoofing attack

 

• Cyberattacks on cloud services
• Adversarial attacks on deep learning models in perception and decision layers

 

DEFENCE METHODS

• Defence against physical sensor attacks
• Defence for cloud services
• Defence against adversarial evasion attacks( Proactive defences, Reactive defence)
• Defence against adversarial poisoning attacks

 

POTENTIAL ATTACKS IN FUTURE

• Adversarial attacks on the whole ADS
• Semantic adversarial attacks
• Reverse-engineering attacks

 

STRATEGIES FOR ROBUSTNESS IMPROVEMENT

• Hardware redundancy
• Model robustness training
• Model testing and verification
• Adversarial attacks detection in real time

 The threats are as below

The paper link is

Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defences

 

Image sources:

Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defences

Source Prolead brokers usa

Why you should put Fixed Point Strategies in Your Toolbox

• Fixed point strategies can approximate infinite depth.
• The methods are easy to train/implement.
• This essential set of tools can model and analyze a wide range of DS problems.

Fixed point theory, which first developed about 60 years ago, is directly connected to limits and traditional control and optimization [1]. These methods are ideal for finding solutions to a broad range of phenomena that crop up in large-scale optimization and highly structured data problems. They work for problems formulated as minimization problems, or more general forms like Nash equilibria [no term] or nonlinear operator equations.  

Compared to traditional models, fixed point methods are in their infancy. However, there’s a lot of research suggesting that these algorithms may be the future of data science.

How do Fixed Point Methods Work?

Fixed point theory works in a similar way to optimization and is related to the idea of a limit in calculus: at some point in the process, you get “close enough” to a solution: one that’s good enough for your purposes. When it’s not possible to find an exact solution, or an exact answer isn’t needed, a fixed-point algorithm can give an approximate answer.

As a simple example, your company might want to create a model for the maximum amount of money a U.S. citizen is willing to spend on new household gadgets per year. An exact solution would depend on many factors, including the fickle nature of consumerism, changing tastes and the effect of climate change on purchasing decisions. It would be difficult (if not impossible) to find an exact solution ($561.23? $981.65?). But there’s going to be a cap, or a limit, which the amount spent tends towards: possibly $570 per annum, possibly $1,000.   

You could attempt to find the solution to a large-scale optimization problem like this one with traditional methods—if you have the computer resources to take on the challenge. In some cases, even the most powerful computer may not be able to hand the computations, which is where fixed point theory steps in.

Advantages of Fixed-Point Methods

Fixed point methods have several major advantages over traditional methods. They create a more efficient framework for implicit depth without requiring more memory or increasing the computational costs of training [2]. On the algorithmic front, fixed point strategies include powerful convergence principles which simplify the design and analysis of iterative methods. In addition, block-coordinate or block-iterative strategies reduce an iteration’s computational load and memory requirements [3].

Google research scientist Zelda Mariet and MIT professor Suvrit Sra approached the problem of maximum-likelihood estimation [no term] by comparing performance of the EM algorithm against a novel fixed-point iteration [4].  When the authors compared performance on both synthetic and real-world data, they found that their fixed-point method gave shorter runtimes when handling large matrices and training sets. The fixed-point approach also ran “remarkably faster” for a range of ground set sizes and number of samples. Not only was it faster than the EM algorithm, but it was also remarkably simple to implement.

The Future of Deep Learning?

One of the major problems with the creation of deep learning models is that the deeper and more expressible a model becomes, the more memory is required.  In a practical sense, the amount of computer memory is limited by model depth. A workaround is implicit depth methods, but these come with the burden of more computational cost to train networks. At a certain point, some problems simply become too complex to be solve using traditional methods. As we go on to the future, models are destined to become more complex, which means we must find better ways to arrive at solutions.

When finding an exact solution isn’t possible because of computational limits, many problems can be formulated in terms of fixed-point optimization schemes.  These schemes, applied to  standard models, guarantee the convergence of a solution to the fixed point limit. 

References:

Image: mikemacmarketing / photo on flickr, CC BY 2.0 , via Wikimedia Commons

[1] Fixed Point Theory and Algorithms for Sciences and Engineering

[2] Fixed Point Networks: Implicit Depth Models with Jacobian-Free Backprop

[3] Fixed Point Strategies in Data Science

[4] Fixed-point algorithms for learning determinantal point processes

 

Source Prolead brokers usa

Honey I Shrunk the Model: Why Big Machine Learning Models Must Go Small

Bigger is not always better for machine learning. Yet, deep learning models and the datasets on which they’re trained keep expanding, as researchers race to outdo one another while chasing state-of-the-art benchmarks. However groundbreaking they are, the consequences of bigger models are severe for both budgets and the environment alike. For example, GPT-3, this summer’s massive, buzzworthy model for natural language processing, reportedly cost $12 million to train. What’s worse, UMass Amherst researchers found that the computing power required to train a large AI model can produce over 600,000 pounds of CO2 emissions – that’s five times the amount of the typical car over its lifespan.

At the pace the machine learning industry is moving today, there are no signs of these compute-intensive efforts slowing down. Research from OpenAI showed that between 2012 and 2018, computing power for deep learning models grew a shocking 300,000x, outpacing Moore’s Law. The problem lies not only in training these algorithms, but also running them in production, or the inference phase. For many teams, practical use of deep learning models remains out of reach, due to sheer cost and resource constraints.

Luckily, researchers have found a number of new ways to shrink deep learning models and optimize training datasets via smarter algorithms, so that models can run faster in production with less computing power. There’s even an entire industry summit dedicated to low-power, or tiny machine learning. Pruning, quantization, and transfer learning are three specific techniques that could democratize machine learning for organizations who don’t have millions of dollars to invest in moving models to production. This is especially important for “edge” use cases, where larger, specialized AI hardware is physically impractical.

The first technique, pruning, has become a popular research topic in the past few years. Highly cited papers including Deep Compression and the Lottery Ticket Hypothesis showed that it’s possible to remove some of the unneeded connections among the “neurons” in a neural network without losing accuracy – effectively making the model much smaller and easier to run on a resource-constrained device. Newer papers have further tested and refined earlier techniques to develop smaller models that achieve even greater speeds and accuracy levels. For some models, like ResNet, it’s possible to prune them by approximately 90 percent without impacting accuracy.

A second optimization technique, quantization, is also gaining popularity. Quantization covers a lot of different techniques to convert larger input values to smaller output values. In other words, running a neural network on hardware can result in millions of multiplication and addition operations. Reducing the complexity of these mathematical operations can help to shrink memory requirements and computational costs, resulting in big performance gains.

Finally, while this isn’t a model-shrinking technique, transfer learning can help in situations where there’s limited data on which to train a new model. Transfer learning uses pre-trained models as a starting point. The model’s knowledge can be “transferred” to a new task using a limited dataset, without having to retrain the original model from scratch. This is an important way to reduce the compute power, energy and money required to train new models.

The key takeaway is that models can (and should) be optimized whenever possible to operate with less computing power. Finding ways to reduce model size and related computing power – without sacrificing performance or accuracy – will be the next great unlock for machine learning.

When more people can run deep learning models in production at lower cost, we’ll truly be able to see new and innovative applications in the real world. These applications can run anywhere – even on the tiniest of devices – at the speed and accuracy needed to make split-second decisions. Perhaps the best effect of smaller models is that the entire industry can lower its environmental impact, instead of increasing it 300,000 times every six years.

Source Prolead brokers usa