Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug/#257 detail 로직 변경 #258

Merged
merged 41 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
ef1e803
Dev be (#185)
swkim12345 Nov 18, 2024
41a4115
Feature/#99 - 참여한 채팅방에서 이전 기록을 확인할 수 있다. (#190)
xjfcnfw3 Nov 19, 2024
8fe29ef
✨ feat: token entity 추가
swkim12345 Nov 25, 2024
903264e
✨ feat: entity 에 저장, expire 검사 로직 추가
swkim12345 Nov 25, 2024
e4429e5
🐛 fix: token 주입으로 로직 변경
swkim12345 Nov 25, 2024
3a81ad0
♻️ refactor: token 주입으로 변경, 그로 인한 오류 수정 및 console.log 삭제
swkim12345 Nov 25, 2024
57f1d5a
🐛 fix: live 데이터 수집 오류 해결, 데이터 없을 때 insert 오류 해결
swkim12345 Nov 25, 2024
1de39b4
🐛 fix: stock, livedata entity 수정
swkim12345 Nov 25, 2024
bb71125
♻️ refactor: develop 환경시 logging 활성화
swkim12345 Nov 25, 2024
e7e6724
📦️ ci: production 환경일 때 작동되게 변경
swkim12345 Nov 25, 2024
8f43995
♻️ refactor: websocket 모듈에서 liveData로 서비스 로직 분리
swkim12345 Nov 25, 2024
6bcb5c5
♻️ refactor: livedata stock module로 이동
swkim12345 Nov 25, 2024
93df868
♻️ refactor: dev-be와 merge
swkim12345 Nov 25, 2024
77e4105
✨ feat: 장 마감시 openapi로 부르는 로직 추가
swkim12345 Nov 25, 2024
fe29d50
🐛 fix: websocket logger 추가, client stock 저장되지 않는 오류 해결
swkim12345 Nov 25, 2024
007e2e5
🐛 fix: openapi는 저장이 필요 없음 롤 백
swkim12345 Nov 25, 2024
2a66928
💄 style: merging dev-be
swkim12345 Nov 25, 2024
ff9ad02
🐛 fix: open api로 데이터 받지 못하는 문제 해결
swkim12345 Nov 25, 2024
ddccc0a
💄 style: 안 쓰이는 것 빼기
swkim12345 Nov 25, 2024
df5b6e6
💄 style: console.log 삭제
swkim12345 Nov 25, 2024
79a34b6
🐛 fix: pk가 아닌 곳에 unique 키 추가
swkim12345 Nov 26, 2024
a751756
💄 style: 불필요한 logger 삭제
swkim12345 Nov 26, 2024
cacfefd
Merge branch 'feature/#240' into bug/#250
swkim12345 Nov 26, 2024
ea8d57e
♻️ refactor: error, disconnect function 분리
swkim12345 Nov 26, 2024
2692830
♻️ refactor: stock data에 indexing, unique 거릭
swkim12345 Nov 26, 2024
df7d087
🐛 fix: 타입 가드 빠진 부분을 추가하고, detail이
swkim12345 Nov 26, 2024
0d3f4c4
🐛 fix: 유량 제어 제거, try-catch로 다시 시작 추가
swkim12345 Nov 26, 2024
390a8b3
♻️ refactor: cron 추가
swkim12345 Nov 26, 2024
4682c57
🐛 fix: cron
swkim12345 Nov 26, 2024
39db490
♻️ refactor: period data 수정
swkim12345 Nov 26, 2024
315db7c
♻️ refactor: console.log 삭제
swkim12345 Nov 26, 2024
04b68c1
♻️ refactor: dev-be 머지
swkim12345 Nov 26, 2024
1429a8a
🐛 fix: token의 expired 먼저 확인하고 db 접근으로 변경
swkim12345 Nov 26, 2024
1f8f82e
🐛 fix: settimeout 시간 조정
swkim12345 Nov 26, 2024
44e54fb
♻️ refactor: 확인용 getItemchartprice 제거
swkim12345 Nov 26, 2024
6e503bc
✨ feat: detail 구현 완료
swkim12345 Nov 26, 2024
129a190
🐛 fix: insert시 데이터 있으면 발생하는 오류 수정
swkim12345 Nov 26, 2024
a1546f8
💄 style: 테스트용 start 삭제, cron만 남겨놓음
swkim12345 Nov 26, 2024
6720a85
♻️ refactor: merge dev-be
swkim12345 Nov 26, 2024
b4ee3d3
🐛 fix: unique column 조건 rollback
swkim12345 Nov 26, 2024
9eccb18
🐛 fix: detail data에 시범적으로 추상클래스 적용
swkim12345 Nov 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions packages/backend/src/scraper/openapi/api/openapi.abstract.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { DataSource } from 'typeorm';
import { openApiConfig } from '../config/openapi.config';
import { Stock } from '@/stock/domain/stock.entity';

export abstract class Openapi {
constructor(protected readonly datasource: DataSource) {}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

혹시 이걸 만드신 이유가 궁금합니다!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

전반적으로 openapi 내부 클래스들이 너무 중구난방이라 추상클래스로 묶은 다음 관리하려고 그랬습니다! 일단 향후 만들 클래스는 추상클래스를 상속받은 다음 구현할 까 합니다.


protected abstract start(): Promise<void>;

protected abstract interval(idx: number, stocks: Stock[]): Promise<void>;

protected abstract step(idx: number, stock: Stock): Promise<void>;

protected abstract getFromUrl(
config: typeof openApiConfig,
stockId: string,
): object;

protected abstract convertResToEntity(res: object, stockId: string): object;

protected async getStockId() {
const entity = Stock;
const manager = this.datasource.manager;
const result = await manager.find(entity, {
select: { id: true },
where: { isTrading: true },
});
return result;
}
}
318 changes: 76 additions & 242 deletions packages/backend/src/scraper/openapi/api/openapiDetailData.api.ts
Original file line number Diff line number Diff line change
@@ -1,278 +1,112 @@
import { Inject, Injectable } from '@nestjs/common';
import { Cron } from '@nestjs/schedule';
import { Between, DataSource } from 'typeorm';
import { DataSource } from 'typeorm';
import { Logger } from 'winston';
import { openApiConfig } from '../config/openapi.config';
import {
DetailDataQuery,
FinancialRatio,
isFinancialRatioData,
isProductDetail,
ProductDetail,
StockDetailQuery,
} from '../type/openapiDetailData.type';
import { TR_IDS } from '../type/openapiUtil.type';
import { DetailData, isDetailData } from '../type/openapiDetailData.type';
import { TR_ID } from '../type/openapiUtil.type';
import { getOpenApi } from '../util/openapiUtil.api';
import { Openapi } from './openapi.abstract';
import { OpenapiTokenApi } from './openapiToken.api';
import { KospiStock } from '@/stock/domain/kospiStock.entity';
import { Stock } from '@/stock/domain/stock.entity';
import { StockDaily } from '@/stock/domain/stockData.entity';
import { StockDetail } from '@/stock/domain/stockDetail.entity';

@Injectable()
export class OpenapiDetailData {
private readonly financialUrl: string =
'/uapi/domestic-stock/v1/finance/financial-ratio';
private readonly productUrl: string =
'/uapi/domestic-stock/v1/quotations/search-stock-info';
private readonly intervals = 1000;
export class OpenapiDetailData extends Openapi {
private readonly TR_ID: TR_ID = 'FHKST01010100';
private readonly url: string =
'/uapi/domestic-stock/v1/quotations/inquire-price';
constructor(
private readonly openApiToken: OpenapiTokenApi,
private readonly datasource: DataSource,
@Inject('winston') private readonly logger: Logger,
protected readonly datasource: DataSource,
private readonly config: OpenapiTokenApi,
) {
//this.getDetailData();
}

@Cron('0 8 * * 1-5')
async getDetailData() {
if (process.env.NODE_ENV !== 'production') return;
const entityManager = this.datasource.manager;
const stocks = await entityManager.find(Stock);
const configCount = (await this.openApiToken.configs()).length;
const chunkSize = Math.ceil(stocks.length / configCount);

for (let i = 0; i < configCount; i++) {
this.logger.info((await this.openApiToken.configs())[i]);
const chunk = stocks.slice(i * chunkSize, (i + 1) * chunkSize);
this.getDetailDataChunk(chunk, (await this.openApiToken.configs())[i]);
}
}

private async saveDetailData(stockDetail: StockDetail) {
const manager = this.datasource.manager;
const entity = StockDetail;
const existingStockDetail = await manager.findOne(entity, {
where: {
stock: { id: stockDetail.stock.id },
},
});
if (existingStockDetail) {
manager.update(
entity,
{ stock: { id: stockDetail.stock.id } },
stockDetail,
);
} else {
manager.save(entity, stockDetail);
}
}

private async saveKospiData(stockDetail: KospiStock) {
const manager = this.datasource.manager;
const entity = KospiStock;
const existingStockDetail = await manager.findOne(entity, {
where: {
stock: { id: stockDetail.stock.id },
},
});

if (existingStockDetail) {
manager.update(
entity,
{ stock: { id: stockDetail.stock.id } },
stockDetail,
);
} else {
manager.save(entity, stockDetail);
super(datasource);
}

@Cron('35 0 * * 1-5')
async start() {
const stock = await this.getStockId();
const len = (await this.config.configs()).length;
const stockSize = Math.ceil(stock.length / len);
let i = 0;
while (i < len) {
this.interval(i, stock.slice(i * stockSize, (i + 1) * stockSize));
i++;
}
}

private async calPer(eps: number): Promise<number> {
if (eps <= 0) return NaN;
const manager = this.datasource.manager;
const latestResult = await manager.find(StockDaily, {
skip: 0,
take: 1,
order: { createdAt: 'desc' },
});
// TODO : price가 없는 경우 0으로 리턴, 나중에 NaN과 대응되게 리턴
if (latestResult && latestResult[0] && latestResult[0].close) {
const currentPrice = latestResult[0].close;
const per = currentPrice / eps;

if (isNaN(per)) return 0;
else return per;
} else {
return 0;
protected async interval(idx: number, stocks: Stock[]) {
const interval = 100;
let time = 0;
for (const stock of stocks) {
setTimeout(() => this.step(idx, stock), time);
time += interval;
}
}

private async calMarketCap(lstg: number) {
const manager = this.datasource.manager;
const latestResult = await manager.find(StockDaily, {
skip: 0,
take: 1,
order: { createdAt: 'desc' },
});

// TODO : price가 없는 경우 0으로 리턴, 나중에 NaN과 대응되게 리턴
if (latestResult && latestResult[0] && latestResult[0].close) {
const currentPrice = latestResult[0].close;
const marketCap = lstg * currentPrice;

if (isNaN(marketCap)) return 0;
else return marketCap;
} else {
return 0;
protected async step(idx: number, stock: Stock) {
try {
const config = (await this.config.configs())[idx];
const res = await this.getFromUrl(config, stock.id);
if (res.output && isDetailData(res.output)) {
const entity = this.convertResToEntity(res.output, stock.id);
await this.save(entity);
}
} catch (error) {
this.logger.warn(`Error in detail data : ${error}`);
setTimeout(() => this.step(idx, stock), 100);
}
}

private async get52WeeksLowHigh() {
const manager = this.datasource.manager;
const nowDate = new Date();
const weeksAgoDate = this.getDate52WeeksAgo();
// 주식의 52주간 일단위 데이터 전체 중에 최고, 최저가를 바탕으로 최저가, 최고가 계산해서 가져오기
const output = await manager.find(StockDaily, {
select: ['low', 'high'],
where: {
startTime: Between(weeksAgoDate, nowDate),
},
});
const result = output.reduce((prev, cur) => {
if (prev.low > cur.low) prev.low = cur.low;
if (prev.high < cur.high) prev.high = cur.high;
return cur;
}, new StockDaily());
let low = 0;
let high = 0;
if (result.low && !isNaN(result.low)) low = result.low;
if (result.high && !isNaN(result.high)) high = result.high;
return { low, high };
protected async getFromUrl(config: typeof openApiConfig, stockId: string) {
const query = this.query(stockId);
const res = await getOpenApi(this.url, config, query, this.TR_ID);
if (res) return res;
else throw new Error();
}

private async makeStockDetailObject(
output1: FinancialRatio,
output2: ProductDetail,
stockId: string,
): Promise<StockDetail> {
protected convertResToEntity(res: DetailData, stockId: string): StockDetail {
const result = new StockDetail();
result.eps = parseInt(res.eps);
result.high52w = parseInt(res.w52_hgpr);
result.low52w = parseInt(res.w52_lwpr);
result.marketCap = res.hts_avls;
result.per = parseFloat(res.per);
Comment on lines +71 to +75
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

오 parseInt, parseFloat 따로 있군요 프론트측에서도 이렇게 받아야겠네요

result.stock = { id: stockId } as Stock;
result.marketCap =
(await this.calMarketCap(parseInt(output2.lstg_stqt))) + '';
result.eps = parseInt(output1.eps);
const { low, high } = await this.get52WeeksLowHigh();
result.low52w = low;
result.high52w = high;
const eps = parseInt(output1.eps);
if (isNaN(eps)) result.eps = 0;
else result.eps = eps;
const per = await this.calPer(eps);
if (isNaN(per)) result.per = 0;
else result.per = per;
result.updatedAt = new Date();
return result;
}

private async makeKospiStockObject(output: ProductDetail, stockId: string) {
const ret = new KospiStock();
ret.isKospi = output.kospi200_item_yn === 'Y' ? true : false;
ret.stock = { id: stockId } as Stock;
return ret;
}

private async getFinancialRatio(stock: Stock, conf: typeof openApiConfig) {
const dataQuery = this.getDetailDataQuery(stock.id!);
// 여기서 가져올 건 eps -> eps와 per 계산하자.
try {
const response = await getOpenApi(
this.financialUrl,
conf,
dataQuery,
TR_IDS.FINANCIAL_DATA,
);
if (response.output && response.output[0]) {
const output1 = response.output;
return output1[0];
}
} catch (error) {
this.logger.warn(error);
}
}

private async getProductData(stock: Stock, conf: typeof openApiConfig) {
const defaultQuery = this.getFinancialDataQuery(stock.id!);

// 여기서 가져올 건 lstg-stqt - 상장주수를 바탕으로 시가총액 계산, kospi200_item_yn 코스피200종목여부 업데이트
try {
const response = await getOpenApi(
this.productUrl,
conf,
defaultQuery,
TR_IDS.PRODUCTION_DETAIL,
);
if (response.output) {
const output2 = response.output;
return output2;
}
} catch (error) {
this.logger.warn(error);
}
}

private async getDetailDataDelay(stock: Stock, conf: typeof openApiConfig) {
const output1 = await this.getFinancialRatio(stock, conf);
const output2 = await this.getProductData(stock, conf);

if (isFinancialRatioData(output1) && isProductDetail(output2)) {
const stockDetail = await this.makeStockDetailObject(
output1,
output2,
stock.id!,
);
this.saveDetailData(stockDetail);
const kospiStock = await this.makeKospiStockObject(output2, stock.id!);
this.saveKospiData(kospiStock);

this.logger.info(`${stock.id!} detail data is saved`);
}
}

private async getDetailDataChunk(chunk: Stock[], conf: typeof openApiConfig) {
let delay = 0;
for await (const stock of chunk) {
setTimeout(() => this.getDetailDataDelay(stock, conf), delay);
delay += this.intervals;
}
}
//private async getStockId() {
// const entity = Stock;
// const manager = this.datasource.manager;
// const result = await manager.find(entity, {
// select: { id: true },
// where: { isTrading: true },
// });
// return result;
//}

private getFinancialDataQuery(
stockId: string,
code: '300' | '301' | '302' | '306' = '300',
): StockDetailQuery {
return {
pdno: stockId,
prdt_type_cd: code,
};
}

private getDetailDataQuery(
stockId: string,
divCode: 'J' = 'J',
classify: '0' | '1' = '0',
): DetailDataQuery {
private async save(saveEntity: StockDetail) {
const entity = StockDetail;
const manager = this.datasource.manager;
await manager
.createQueryBuilder()
.insert()
.into(entity)
.values(saveEntity)
.orUpdate(
['market_cap', 'eps', 'per', 'high52w', 'low52w', 'updated_at'],
['stock_id'],
)
.execute();
}

protected query(stockId: string, code: 'J' = 'J') {
return {
fid_div_cls_code: classify,
fid_cond_mrkt_div_code: divCode,
fid_cond_mrkt_div_code: code,
fid_input_iscd: stockId,
};
}

private getDate52WeeksAgo(): Date {
const today = new Date();
const weeksAgo = 52 * 7;
const date52WeeksAgo = new Date(today.setDate(today.getDate() - weeksAgo));
date52WeeksAgo.setHours(0, 0, 0, 0);
return date52WeeksAgo;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ export class OpenapiLiveData {
}
}

// 현재가 체결로는 데이터가 부족해 현재가 시세를 사용함.
// 현재가 체결
convertResponseToStockLiveData = (
data: OpenapiLiveData,
stockId: string,
Expand Down
Loading