Basics of Oracle Database Architecture

Oracle Database architecture consists of two main components: the database and the instance. Here's a brief overview of each component and their subcomponents: *1. Database:* The database is the physical storage component of the Oracle architecture. It contains datafiles, redo log files, and control files. - Datafiles store user data and database objects like tables, indexes, and clusters. - Redo log files record changes made to the database, allowing for recovery in case of failures. - Control files store the database's physical structure, such as tablespaces, datafiles, and redo log files. *2. Instance:* The instance is the memory and processes component of the Oracle architecture. It consists of the System Global Area (SGA) and background processes. - System Global Area (SGA): This is a shared memory area that stores data and control information for the database instance. The SGA includes the database buffer cache, shared pool, redo log buffer, and other memory structures. - Background processes: These are processes that run in the background, managing various tasks, such as managing database memory structures (DBWn), writing redo log entries (LGWR), monitoring and archiving redo log files (ARCH), and handling other tasks (CKPT, PMON, SMON, RECO). When a user connects to an Oracle database, the instance retrieves data from datafiles and caches it in the SGA. The user's actions generate redo log entries, which are used for recovery if needed. Background processes manage resources and ensure the database functions smoothly.

What is SCN in Oracle dB

SCN stands for System Change Number in the Oracle database. It is a logical, internal timestamp used to order events within the database and ensure the ACID properties of transactions. SCNs help mark the point in time before which all changes are known to be on disk, allowing for efficient recovery and avoiding unnecessary redo operations. Essentially, SCNs act as a clock for the database, incrementing every time a COMMIT is made. They are also used in FLASHBACK queries and to restore data to a specific point in time. In the Oracle database architecture, the System Change Number (SCN) plays a crucial role in maintaining data consistency, concurrency, and recoverability. Here are some key details on the Oracle architecture and SCN: *Oracle Architecture*: The Oracle database system is divided into the database (physical storage) and the instance (memory and processes). Multiple instances can access the same database, but an instance can only access one database at a time. The database contains datafiles, redo log files, and control files, while the instance includes the System Global Area (SGA) and background processes. *SCN*: SCN is a logical timestamp used to track changes in the database and maintain the order of events. It helps in achieving the ACID properties of transactions. SCN is incremented each time a COMMIT occurs and is used in various scenarios, such as: - *Consistency*: SCN helps maintain data consistency by ensuring that a read operation retrieves data from a consistent point in time, using the SCN to reference the required version of data blocks. - *Recovery*: During database recovery, SCN helps identify the point to which the database needs to be restored, minimizing data loss and ensuring database consistency. - *Flashback Queries*: SCN is used to view data as it appeared at a specific point in time, enabling users to retrieve data that may have been accidentally deleted or modified. In summary, SCN is a critical component of the Oracle database architecture, ensuring the reliability and consistency of the database through its role in transaction management, recovery, and data consistency.

5S

SORT (SEIRI) - Sort out your stuff SET-IN-ORDER (SEITON) - Set things in order so that we can find it easily SHINE (SEISO) - Keep workplace clean and identify the source of sort STANDARDIZE (SEIKETSU) - create rules, practices for consistency in clean and order SUSTAIN (SHITSUKE) -

Intec Singl.eView Billing Fundamentals

Intec Singl.eView Billing Fundamentals: Singl.eview three tier Architecture: Tier1 • Tier 1 is the interface of the ’outside’ world with Convergent Billing. Clients, remote hosts, and external applications enter, inspect, modify data, and initiate processes in Convergent Billing. Tier2 • Tier 2 consists of Convergent Billing’s expression-driven configuration, the Transaction Engine (TRE), and non-TRE Convergent Billing processes. Tier 2 manages transactions initiated from, and returns results to, the first tier, as well as managing all database access. Supported platforms for Convergent Billing are AIX, Solaris, and HP-UX (PA-RISC and IA64). Tier3 • The database tier in Convergent Billing consists of the Oracle database, which stores business, administrative, and configuration data used in Convergent Billing. Singl.eView Configuration Layers: Configuration elements are: • Convergent Billing core software, consisting of key functionality for customer care, rating, billing, and workflow. • Common framework processes, interfaces, and subsystems, consisting of a set of predefined solutions for common configuration requirements. • Market-specific solutions and preconfigured product sets, consisting of configuration that is common to a specific set of products or solution. • Client-specific business rules, consisting of a collection of billing rules that the service provider configures to align customer care and billing processing to reflect their own business requirements. Rating and Billing Overview: Rating and billing is divided into seven operations: • Balance management, which provides event authorization, advice of charge information, and customer credit reservations. • Event normalization, in which input event and transaction data is converted into a standard format for storage in the Convergent Billing database. • Event rating, in which the normalized event records are aggregated and rated (costed) by applying the appropriate tariffs. • Event output, in which the costed event records are stored in the Convergent Billing database. • Billing, in which the costed event records are aggregated, and additional charges and discounts can be applied (for example, for recurring events). • Invoicing, in which billing data is combined with customer detail records and incorporated into an invoice or statement image. • Invoice output, in which invoice and statement images are printed or converted for electronic distribution. The seven operations are carried out by the following three major components: • The trerate TRE server, which authorizes real-time events, handles credit checking, and passes completed events on to the rating engine to be stored in the database. • The rating engine, which combines normalization and rating. The rating engine is a series of pipelined processes, which primarily communicate using shared memory. Rating input is raw call (or ’event’) records; output is database records containing normalized events and charges rated against the events (costed events). • The billing engine, which passes information back to the rating engine for the generation of recurring charges and adjustments, calculates billing charges, generates invoice data, and creates the invoice images for printing. Billing operations include the selection, sorting, and output of invoices or statements to printers or other output devices. TRE Overview and Functionality: The Transaction Engine (TRE) forms the core of the application tier of the open, three-tier architecture of Convergent Billing that is based on Tuxedo middleware. The TRE provides: • Transaction management • Cache management • Connection authentication • Multiple client access • Asynchronous alerts • Comprehensive set of business services and functions (exposed using APIs). Convergent Billing Interfaces : Convergent Billing is able to interface in multiple ways with multiple external systems, including the following: • Siebel • Oracle Financials • SAP • Clarify • PeopleSoft • Various network switches in batch and real-time modes. Convergent Billing Data Model : The Convergent Billing data model has the following points: • Entity relationships are determined dynamically through the configured business rules. • Historical information is maintained for all entities (through the use of date ranging). • Interpretation of data in generic table columns is dependent upon configuration. The main database entities in the Convergent Billing data model are: • Accounts • Contacts • Customers • Payments • Products • Queries. Customizing Convergent Billing : • Reference Types Reference types are defined lists of items, and are referenced throughout Convergent Billing; many of the drop-down lists available on the customer care forms are defined as reference types. Reference types are associated with attribute types to include drop-down lists in new and customized fields. • Attribute Types Attribute types are the major building blocks for entity validation and define the attributes of a field. It overrides the field’s existing attributes, allowing the field to be customised. • Entity Validation Entity validation is the key to extensively configuring the customer care environment. Entity validation is used to customize and add fields to customer care forms, and add validation. Both attribute types and reference types are used in the definition of entity validation. • Derived Attributes Derived attribute variables return one or more values based either on an expression or a lookup of a table defined within the derived attribute. Derived attribute variables and derived attribute tables allow service providers to capture and store their business rules in a tabular format. Derived attributes are variables derived from one or more other variables, by using: • Simple expressions • Conditional evaluation (similar to functions) • Lists (tables) • Table look-ups. Product Data Model : Product Model Entities Entities that comprise the product data model are described in the following sections. • Equipment A product may have one or more pieces of equipment. Equipment can be a physical piece of equipment (for example, a telephone) or conceptual (for example, a telephone number). Equipment can be reused for allocation to multiple services. • Service A product must have at least one service. For example, a wire line or wireless telephony product might offer one or more different service types including voice or fax. • Facility Groups A facility group is a collection of one or more service options (also referred to as facilities, features, supplementary services, or value-add services). The group is associated with a specified service and defined as part of a product. • Product Groups Products can be grouped to allow an operator to locate related products more easily when assigning products to customers. • Tariffs Tariffs specify the charges and benefits applied to events, and the rules for applying them, to calculate cost information for inclusion on an invoice or statement. Tariffs can be used to determine: • Whether to apply a charge • Amount of the charge • Allocation of the charge to the appropriate accounts and GL • codes. Charge Categories : • A charge category identifies the default account type and general ledger (GL) code to which a calculated tariff charge or benefit is allocated. For services, the charge category also identifies a specific account number. The association between a tariff and a charge category is specified when the tariff is defined, and is referred to as the tariff/charge category pair. • Charge categories allow guiding to a ’To’ account, ’From’ account, ’To’ GL code, and ’From’ GL code. The account type specified in a charge category definition is translated to the actual account number and stored in the charge category instance (which is created when the product instance is created). Batch Rating Engine : Steps Used by Rating Engine The basic steps of input, normalizing, rating, and output are used by the rating engine and outlined in the following steps. • Normalising Before Convergent Billing can rate events, incoming events must first be normalised. Normalising converts an event to the Convergent Billing native event format and is executed by the Event Normalisation (ENM) process. The normalisation process validates the record, verifies its accuracy, and formats it for rating. Convergent Billing provides a mapping language called DIL (Data Interface Language) that allows event data in any incoming format to be mapped and converted into Convergent Billing normalised events. DIL can handle the mapping of events in both ASCII and binary formats. • Rating During rating, chargeable elements are used by the Event Rating (ERT) process to determine the tariff to be applied and calculated to derive an event charge. Output • After charges are generated by the ERT, the Event Rating (ERO) process takes and outputs the charges and associated normalised events from the event cache. Rating Processes There are basically five billing components that execute the rating functions in Singl.eView. They are as below: • Event Rating Broker (BKR) process • Event Normalisation (ENM) process • Event Rating (ERT) process • Event Rating Output (ERO) process • TRE Rating Server (trerate).

Big Data Learn Step by Step

https://www.mltut.com/how-to-learn-big-data-step-by-step/

REST API

REpresentational State Transfer Application Programming Interface. refers to a group of software architecture design constraints that bring about efficient, reliable, and scalable systems. it is a data architecture and design methodology that produces predicatable and consistent behaviours and ouputs by receiving a set of standard methods called verbs and returning standarized structured data, typically JSON or XML, called the resource. API is a set of features and rules that exist inside of a software program enabling interaction between the software and other items, such as other software or hardware. In the context of REST APIs, The API is the collection of tools used to access and work with REST resources through your adverbs including get, pulls,put, and delete. URL vs URI vs URN ------------------ URL --> Actual physical location. URI vs URL --> All URLs are URIs and not All URIs are URLs. URN --> is a subset of URI. Unique name identifier , say a person. A URN can also be a URL, but doesn't have to be . So in conclusion a URL might also be a URN and both are URIs. The Six Constraints of REST --------------------------- 1) Client-Server Architecture : The client manages user interface concerns while the server manages data storage concerns. 2) Statelessness : No client context or information , aka "state" , can be stored on the server between requests. 3) Cacheability : All REST responses must be clearly marked as cacheable or not cacheable. 4) Layered System: The client cannot know , and shouldn't care, whether it's connected directly to the server or to an intermediary like a CDN or mirror. 5) Code on Demand: servers are allowed to transfer executable code like client side JavaScript and compiled components to clients. 6) Uniform Interface: 6.1 : Resource identification in requests: The URI request must specify what resource it is looking for and what format thte response should use. 6.2 : Resource manipulation through representations: Once a client has a representation of a resource, it can modify or delete the resource. 6.3 : Self-descriptive messages: A unform interface must issue self-descriptive messages. This goes to both sending and receiving REST data. Each representation must describe its own data format. 6.4 : Hypermedia as the engine of application state: Once a client has access to a REST service, it should be able to discover al available resources and methods through the hyperlinks provided. Humans are not directly interacting with the REST API, Communication with the REST API is handled by the client, which can be anything , really. A website, an app, even an internet of things device.

Python Script for Data Recon

from collections import OrderedDict as od import pandas as pd def diff_func(df_left, df_right, uid, labels=('Left', 'Right'), drop=[[],[]]): dict_df = {labels[0]: df_left, labels[1]: df_right} col_left = df_left.columns.tolist() col_right = df_right.columns.tolist() # There could be columns known to be different, hence allow user to pass this as a list to be dropped. if drop[0] or drop[1]: print ('{}: Ignoring columns {} in comparison.'.format(labels[0], ', '.join(drop[0]))) print ('{}: Ignoring columns {} in comparison.'.format(labels[1], ', '.join(drop[1]))) col_left = list(filter(lambda x: x not in drop[0], col_left)) col_right = list(filter(lambda x: x not in drop[1], col_right)) df_left = df_left[col_left] df_right = df_right[col_right] # Step 1 - Check if no. of columns are the same: len_lr = len(col_left), len(col_right) assert len_lr[0]==len_lr[1], \ 'Cannot compare frames with different number of columns: {}.'.format(len_lr) # Step 2a - Check if the set of column headers are the same # (order doesnt matter) assert set(col_left)==set(col_right), \ 'Left column headers are different from right column headers.' \ +'\n Left orphans: {}'.format(list(set(col_left)-set(col_right))) \ +'\n Right orphans: {}'.format(list(set(col_right)-set(col_left))) # Step 2b - Check if the column headers are in the same order if col_left != col_right: print ('[Note] Reordering right Dataframe...') df_right = df_right[col_left] # Step 3 - Check datatype are the same [Order is important] if all(df_left.dtypes == df_right.dtypes): print ('DataType check: Passed') else: print ('dtypes are not the same.') df_dtypes = pd.DataFrame({labels[0]:df_left.dtypes,labels[1]:df_right.dtypes,'Diff':(df_left.dtypes == df_right.dtypes)}) df_dtypes = df_dtypes[df_dtypes['Diff']==False][[labels[0],labels[1],'Diff']] print (df_dtypes) # Step 4 - Check for duplicate rows for key, df in dict_df.items(): if df.shape[0] != df.drop_duplicates().shape[0]: print(key + ': Duplicates exists, they will be dropped.') dict_df[key] = df.drop_duplicates() # Step 5 - Check for duplicate uids. if isinstance(uid, (str, list)): print ('Uniqueness check: {}'.format(uid)) for key, df in dict_df.items(): count_uid = df.shape[0] count_uid_unique = df[uid].drop_duplicates().shape[0] dp = [0,1][count_uid_unique == df.shape[0]] #<-- Round off to the nearest integer if it is 100% pct = round(100*count_uid_unique/df.shape[0], dp) print ('{}: {} out of {} are unique ({}%).'.format(key, count_uid_unique, count_uid, pct)) # Checks complete, begin merge. d_result = od() d_result[labels[0]] = df_left d_result[labels[1]] = df_right if all(df_left.eq(df_right).all()): print('Trival case: DataFrames are an exact match.') d_result['Merge'] = df_left.copy() else: df_merge = pd.merge(df_left, df_right, on=col_left, how='inner') if not df_merge.shape[0]: print('Trival case: Merged DataFrame is empty') d_result['Merge'] = df_merge if type(uid)==str: uid = [uid] if type(uid)==list: df_left_only = df_left.append(df_merge).reset_index(drop=True) df_left_only['Duplicated']=df_left_only.duplicated(keep=False) #keep=False, marks all duplicates as True df_left_only = df_left_only[~df_left_only['Duplicated']] df_right_only = df_right.append(df_merge).reset_index(drop=True) df_right_only['Duplicated']=df_right_only.duplicated(keep=False) df_right_only = df_right_only[~df_right_only['Duplicated']] label = '{} or {}'.format(*labels) df_lc = df_left_only.copy() df_lc[label] = labels[0] df_rc = df_right_only.copy() df_rc[label] = labels[1] df_c = df_lc.append(df_rc).reset_index(drop=True) df_c['Duplicated'] = df_c.duplicated(subset=uid, keep=False) df_c1 = df_c[df_c['Duplicated']] df_c1 = df_c1.drop('Duplicated', axis=1) cols = df_c1.columns.tolist() df_c1 = df_c1[[cols[-1]]+cols[:-1]] df_uc = df_c[~df_c['Duplicated']] df_uc_left = df_uc[df_uc[label]==labels[0]] df_uc_right = df_uc[df_uc[label]==labels[1]] d_result[labels[0]+'_only'] = df_uc_left.drop(['Duplicated', label], axis=1) d_result[labels[1]+'_only'] = df_uc_right.drop(['Duplicated', label], axis=1) d_result['Diff'] = df_c1.sort_values(uid).reset_index(drop=True) return d_result